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ABSTRACT 

Using an approach to classroom research that D. 

Newman (1990) has termed a formative experiment, a study investigated 
the effects of engaging elementary school students in creating 
computer-based multimedia reviews of books they read independently. 
Formative experiments are designed to investigate how an 
instructional intervention can be implemented to achieve a 
pedagogical goal in a particular educational environment. Creating 
multimedia book reviews was the intervention; increasing the amount 
and diversity of students' independent reading was the pedagogical 
goal. Diverse quantitative and qualitative data were gathered during 
2 academic years in 9 4th-grade and 5th~grade classrooms across 3 
schools. Consistent with the intent of formative experiments, results 
are presented guided by the following questions: (1) What factors in 

the educational environment enhance or inhibit the intervention's 
effectiveness in achieving the pedagogical goal?; (2) How can the 
intervention and its implementation be modified during the experiment 
to achieve more effectively the pedagogical goal?; (3) What 
unanticipated positive or negative effects does the intervention 
produce?; and (4) Has the educational environment changed as a result 
of the intervention? Results indicated that the multimedia book 
review activity contributed to achieving the pedagogical goal of 
increasing the amount of children's independent reading; and school 
environments and teachers’ roles to some extent shaped the effects of 
the activity. Findings suggest that formative experiments can address 
the limitations of conventional research methods previously used to 
study computer-based literacy activities in classrooms. (Contains 64 
references, and 4 tables and 11 figures of data. Appendixes presents 
parent and teacher questionnaires.) (Author/RS) 
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About the National Reading Research Center 



The National Reading Research Center (NRRC) is 
funded by the Office of Educational Research and 
Improvement of the U.S. Department of Education to 
conduct research on reading and reading instruction. 
The NRRC is operated by a consortium of the Univer- 
sity of Georgia and the University of Maryland College 
Park in collaboration with researchers at several institu- 
tions nationwide. 

The NRRC’s mission is to discover and document 
those conditions in homes, schools, and communities 
that encourage children to become skilled, enthusiastic, 
lifelong readers. NRRC researchers are committed to 
advancing the development of instructional programs 
sensitive to the cognitive, sociocultural, and motiva- 
tional factors that affect children’s success in reading. 
NRRC researchers from a variety of disciplines conduct 
studies with teachers and students from widely diverse 
cultural and socioeconomic backgrounds in pre-kinder- 
garten through grade 12 classrooms. Research projects 
deal with the influence of family and family-school 
interactions on the development of literacy; the interac- 
tion of sociocultural factors and motivation to read; the 
impact of literature -based reading programs on reading 
achievement; the effects of reading strategies instruction 
on comprehension and critical thinking in literature, 
science, and history; the influence of innovative group 
participation structures on motivation and learning; the 
potential of computer technology to enhance literacy; 
and the development of methods and standards for 
alternative literacy assessments. 

The NRRC is further committed to the participation 
of teachers as full partners in its research. A better 
understanding of how teachers view the development of 
literacy, how they use knowledge from research, and 
how they approach change in the classroom is crucial to 
improving instruction. To further this understanding, 
the NRRC conducts school-based research in which 
teachers explore their own philosophical and pedagogi- 
cal orientations and trace their professional growth. 



Dissemination is an important feature of NRRC 
activities. Information on NRRC research appears in 
several formats. Research Reports communicate the 
results of original research or synthesize the findings of 
several lines of inquiry. They are written primarily for 
researchers studying various areas of reading and 
reading instruction. The Perspective Series presents a 
wide range of publications, from calls for research and 
commentary on research and practice to first-person 
accounts of experiences in schools. Instructional 
Resources include curriculum materials, instructional 
guides, and materials for professional growth, designed 
primarily for teachers. 

For more information about the NRRC’s research 
projects and other activities, or to have your name 
added to the mailing list, please contact: 

Donna E. Alvermann, Co-Director 
National Reading Research Center 
318 Aderhold Hall 
University of Georgia 
Athens, GA 30602-7125 
(706) 542-3674 

John T. Guthrie, Co-Director 
National Reading Research Center 
3216 J. M. Patterson Building 
University of Maryland 
College Park, MD 20742 
(301) 405-8035 
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Abstract. Using an approach to classroom 
research that Newman (1990) has termed a 
formative experiment , this study investigated the 
effects of engaging elementary school students in 
creating computer-based multimedia reviews of 
books they read independently. Formative experi- 
ments are designed to investigate how an instruc- 
tional intervention can be implemented to achieve a 
valued pedagogical goal in a particular educational 
environment. In the present study, creating multime- 
dia book reviews was the intervention; increasing 
the amount and diversity of students’ independent 
reading was the pedagogical goal. Diverse quantita- 
tive and qualitative data were gathered during 2 
academic years in 9 fourth- and fifth-grade class- 
rooms across 3 schools. Consistent with the intent of 
formative experiments ; the authors present results 
guided by the following questions: (a) What factors 
in the educational environment enhance or inhibit 
the intervention ’s effectiveness in achieving the 
pedagogical goal?; (b) How can the intervention 
and its implementation be modified during the 
experiment to achieve more effectively the pedagogi- 
cal goal?; (c) What unanticipated positive or nega- 
tive effects does the intervention produce?; and 
(d) Has the educational environment changed as a 



result of the intervention ? In addition to describing 
the study and reporting results , the authors discuss 
emerging understandings of formative experiments. 
They argue that formative experiments can address 
the limitations of conventional research methods 
previously used to study computer-based literacy 
activities in classrooms. 

Diverse approaches to research have been 
used to study whether computer-based instruc- 
tional activities can enhance literacy in 
classrooms. However, most studies have 
used conventional experimental designs 
aimed at comparing the effectiveness of 
interventions with and without computer- 
based activities (cf. Means et al., 1993; Rein- 
king & Pickle, 1993). Despite the prevalence 
of experimental studies, reviews of classroom 
research (e.g., Reinking & Bridwell-Bowles, 
1991) as well as the observations and findings 
of some researchers (Bruce & Rubin, 1993; 
Dickinson, 1986; Mehan, 1989, Michaels & 
Bruce, 1989; Reinking & Pickle, 1993; Rubin 
& Bruce, 1990) suggest that conventional 
experiments alone do not provide adequate 
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information about how computer-based activi- 
ties might effectively contribute to the acquisi- 
tion of literacy in schools. 

Conventional experiments do not adequately 
deal with the many interacting variables that 
influence the effectiveness of computer-based 
interventions in schools; nor do they typically 
deal with how a particular computer-based 
activity might produce unique effects depending 
on how it is integrated into a particular instruc- 
tional environment. Instead, conventional 
experiments require researchers (a) to control 
the influence of most situational factors; (b) to 
implement a well-defined intervention that 
remains unchanged during the experiment, 
often in the face of changing conditions in the 
classroom; and (c) to focus primarily on post- 
intervention outcomes. These requirements 
may perpetuate a point of view that restricts 
the influence of technology in schools. As 
Emihovich and Wager (1992) state, 

. . . media are still perceived as add-ons 
to the educational process . . . [and] this 
perception will not change until educators 
begin to realize that media or technology 
use in schools should be examined from 
a holistic cultural perspective. That is, 
the introduction of any new technology 
should be considered in relation to its 
effect on the school culture as a whole, 
including the way students and teachers 
perceive the new technology, how admin- 
istrators view it in relation to the organi- 
zational climate of the school, and the 
expectations parents and the broader 
community share concerning the effects 
of new technologies have, or should 
have, on learning, (pp. 435-36) 



The limited scope of conventional experi- 
mental research involving computers in class- 
rooms may explain in part why that research 
has not produced clear recommendations for 
educators interested in how technology might 
enhance literacy instruction in schools. Indeed, 
that research does not consistently support the 
use of computer-based activities over other 
interventions not using computers (Reinking & 
Bridwell-Bowles, 1991; Roblyer, Castine, & 
King, 1988). As Becker (1992) states in his 
review of the computer-based integrated learn- 
ing systems, 



... the widely varying effects sizes and 
the modestly positive effect sizes that are 
typical suggest that . . . results differ as 
much based on the different conditions of 
the study as on the different software 
packages in use (and on the different 
methodologies used to design and con- 
duct the analysis), (p. 38) 



The relatively few qualitative and ethno- 
graphic studies investigating computers and 
literacy counter some of these limitations by 
focusing on the influence of situational factors 
over time within a particular classroom or 
school (Dickinson, 1986; Friedman, 1990; 
McGee, 1987; Mehan, 1989; Riel, 1989; 
Turner & Dipinto, 1992). However, such 
studies typically do not focus on the most 
instructionally relevant questions: What 
factors add to or detract from an intervention’s 
success in accomplishing a pedagogically 
valued goal and, more importantly, how might 
the intervention be adapted in response to 
those factors? 
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To address the inherent limitations of 
conventional approaches to classroom re- 
search involving computer-based activities 
and to address these instructionally relevant 
questions, we adopted an approach to class- 
room research that Newman (1990) has 
termed a “formative experiment.” Formative 
experiments, as we will describe more fully 
in a subsequent section, are aimed at investi- 
gating how instructional interventions can be 
adapted in response to factors that enhance 
or inhibit their effectiveness in achieving a 
valued pedagogical goal. And, they allow 
researchers to examine a broad range of 
interacting factors and events that influence 
an intervention’s effectiveness as well as its 
unanticipated consequences. The instruc- 
tional intervention reported here involved 
fourth- and fifth-grade students and their 
teachers in creating multimedia book reviews 
related to their independent reading as an 
alternative to the conventional required book 
report. That is, we were interested in sys- 
tematically examining how we might engage 
elementary school students and their teachers 
in creating multimedia book reviews with the 
goal of increasing the amount and diversity 
of students’ independent reading. Our study 
encompassed seven classrooms in three 
schools during two academic years. 

Formative experiments are especially 
applicable to conducting classroom research 
aimed at investigating computer-based inter- 
ventions (Newman, 1990) because the ex- 
pected advantages of such interventions have 
been difficult to achieve. On one hand, it is 
clear that much of the interest in educational 
uses of computers is related to the belief that 



they have strong potential to transform 
positively the standard modes of teaching 
and learning in schools (e.g., Newman, 
1990; Papert, 1993; Sheingold, 1991). On 
the other hand, it is clear from a variety of 
sources that simply introducing innovative, 
powerful, computer-based activities into a 
classroom is typically not enough to achieve 
this potential (Means et al., 1993). Empiri- 
cal studies of how computers are used in 
schools support this conclusion. Such studies 
have documented consistently that the num- 
ber of computers for instruction in schools 
has increased geometrically since the early 
1980s, but the amount of time they are used 
and their level of integration into the curric- 
ulum has not kept pace with availability 
(c.f. , Becker, 1990, 1992a; Congressional 
Office of Technology Assessment, 1988; 
Martinez & Mead, 1988). These findings 
reflect what many writers argue is an inordi- 
nate concern for acquiring technology with- 
out concomitant attention to establishing 
clear pedagogical goals that it’s acquisition 
will promote (see Reeves, 1992). Even when 
schools and teachers attempt to integrate 
computers into instruction to achieve clearly 
defined goals, they often face many obsta- 
cles (Gustafson, 1993; Hadley & Sheingold, 
1993). These obstacles may be technologi- 
cal, logistical, curricular, financial, interper- 
sonal, and so forth. 

Consequently, there has been an increas- 
ingly clear realization based on research and 
supported by the experiences of many educa- 
tors in schools that simply placing computers 
into a school or classroom rarely produces 
notable changes in teaching and learning 
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(see Miller & Olson, 1994). As Bresler and 
Walker (1990) state, “Even when an innova- 
tion meets people’s expressed needs, it may 
still not succeed unless it fits the patterns by 
which they run their lives as students and 
teachers” (p. 66). Yet, many writers and 
educators remain steadfast in their belief that 
computers have strong potential to do so. 

Even when computer-based activities are 
specifically designed to transform or reorient 
teaching and learning, they may be imple- 
mented in such a way as to maintain the 
status quo. A well-documented example is 
the work of Bruce and his colleagues (Bruce 
& Peyton, 1990; Bruce & Rubin, 1993; 
Michaels & Bruce, 1989) who have con- 
ducted extensive investigations of a computer- 
based intervention called QUILL. QUILL is 
a comprehensive computer-based tool to 
support writing and reading. It is comprised 
of distinct components such as the “mailbag” 
(to facilitate on-line communication among 
students and teachers), the “planner” (for 
getting ideas for writing and for organizing 
ideas respective to various genres), the 
“library” (a textual information base created 
by and shared among students and teachers), 
and the “writers assistant” (a word process- 
ing program designed to be compatible with 
the other components). 

Bruce and his colleagues developed 
QUILL specifically to promote valued peda- 
gogical goals in literacy such as critical 
thinking, integration of reading and writing, 
meaningful communication, collaboration, 
and revision. QUILL was also developed 
to be open ended with the intent that it 
would be used differently in each situation. 



For example, some teachers and students 
might use it to develop a classroom news- 
paper while others might focus on writing 
about academic topics. In short, Bruce and 
his colleagues designed QUILL to effect 
changes in the way reading and writing 
were typically carried out in individual 
classrooms. As Michaels and Bruce (1989) 
state, they hoped that it “would alter the 
writing systems, and hence students’ access 
to writing and reading opportunities” (p. 12). 
However, they found that “rather than the 
new technology radically reshaping the 
learning environment, the computers them- 
selves were shaped to fit the already estab- 
lished patterns of social organization” (p. 
12). Based on their analysis of QUILL in 
two classrooms over two years they con- 
cluded that: 

... it becomes clear that no innova- 
tion, no matter how well conceived, 
and no application of innovations, no 
matter how well intended and exe- 
cuted, can in and of themselves be 
assured of achieving positive change in 
instruction. Instead, one must continu- 
ally reexamine what is happening and 
ask critical questions: What is the 
innovation we are using? What are the 
different ways that students are en- 
gaged in learning? What is the effect 
of my actions on students’ learning? 
What are the goals of instruction, of 
each activity? And so on. (Michaels & 
Bruce, 1989, p. 35) 

In a similar vein, Bruce and Rubin (1993) 
state that their “detailed, self-critical ap- 
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praisal of the evidence [about QUILL] yields 
surprises and reveals a richness in what 
students and teachers do that belies both 
optimistic and pessimistic visions of technol- 
ogy in relation to educational change” (p. 
1). Our decision to use a formative experi- 
ment as an approach to research in the pres- 
ent study was based on our concurrence with 
their position and with the position of re- 
searchers who have discussed the limitations 
of conventional experiments in conducting 
classroom research (e.g., Reinking & Pickle, 
1993). 

Formative Experiments 

Dissatisfaction with conventional experiments 
as an approach to classroom research as ex- 
pressed by many writers interested in technol- 
ogy (e.g., Bruce & Rubin 1993; Newman, 
1990; Papert, 1987; Reinking & Bridwell- 
Bowles, 1991; Reinking & Pickle, 1993; 
Venezky, 1983) reflects the broader concern 
that researchers must seek out alternative 
approaches to research that address the com- 
plexity of classrooms (see Eisenhart & Borko, 
1993; Jackson, 1990). The concept of a forma- 
tive experiment as described by Newman 
(1990) is one proposed alternative to conven- 
tional experiments. 

However, it is important to note that the 
concept of a formative experiment is evolving. 
This characterization is evidenced first by the 
fact that there are few published sources of 
information discussing explicitly the concept of 
a formative experiment (to our knowledge 
limited to Newman’s 1990 article in the Educa- 
tional Researcher and our own published work 



based on our experiences in the present study; 
see Reinking & Pickle, 1993; Baumann, Dil- 
lon, Shockley, Alvermann, & Reinking, 1996). 
Second, the concept of a formative experiment 
is not yet clearly distinguishable from related 
concepts such as “situated evaluation” (Bruce 
& Rubin, 1993), “design experiments” 
(Brown, 1992), “formative evaluation” (see 
Flagg, 1990), and “rapid prototyping” (Tripp 
& Bichelmeyer, 1990). While supporting the 
fact that classroom research is evolving, Eisen- 
hart and Borko (1993) state that 

the standards for using ... alternative 
methodologies in educational research 
are not routinized in the same way they 
are for established methodologies; thus 
their use demands more thought and 
explanation than might be necessary if 
conventional procedures were used. 

(p. 11) 

To address this issue, we provide here an 
overview of our evolving understanding of 
formative experiments based on Newman’s 
(1990) description and our own experience 
in the present research project. Thus, the 
present report has a dual purpose. Foremost, 
it provides a description of a computer-based 
intervention and how it was implemented to 
accomplish the goal of increasing the amount 
and diversity of students’ independent reading. 
However, it also reports our expanded under- 
standing of formative experiments and their 
potential to contribute to literacy research. 

Newman (1990) has described a formative 
experiment as follows: “In a formative experi- 
ment, the researcher sets a pedagogical goal 
and finds out what it takes in terms of materials, 
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Conventional Experiments 


Qualitative or Ethonographic 
Studies 


Formative Experiments 


How does the use of multimedia 
book reviews compare to some 
other classroom activity aimed at 
increasing independent reading? 


In a classroom using multimedia 
book reviews (perhaps when 
compared to classrooms using 
alternative activities), what are 
the dominant factors that explain 
why some students read more and 
some read less? 


Given a pedagogical rationale for 
believing that multimedia book 
reviews have potential to meet the 
goal of increasing independent 
reading, how must this activity be 
implemented in a particular class- 
room to achieve its stated goal? 



Figure 1. Matching related research questions and approaches to research. 



organization, or changes in . . . technology to 
reach the goal” (p. 10). Thus, formative exper- 
iments address a different category of questions 
when compared to conventional experiments or 
qualitative approaches to research. Figure 1 
compares how three approaches to research 
can be matched to three related but different 
research questions that might guide an investi- 
gation of multimedia book reviews as a class- 
room intervention. 

Drawing on Newman’s (1990) explication 
of formative experiments and our own experi- 
ence in conducting the present investigation, 
we have proposed six questions as a framework 
for designing and conducting formative experi- 
ments (see Baumann et al., 1996): 

1. What is the pedagogical goal of the 
experiment and what pedagogical theory 
establishes its value? 

2. What is an instructional intervention that 
has potential to achieve the identified 
pedagogical goal? 



3. As the intervention is implemented, what 
factors enhance or inhibit its effectiveness 
in achieving the pedagogical goal? 

4. How can the intervention and its imple- 
mentation be modified to achieve more 
effectively the pedagogical goal? 

5. What unanticipated positive or negative 
effects does the intervention produce? 

6. Has the instructional environment changed 
as a result of the intervention? 

These questions serve as a guide for 
reporting our investigation. Questions 1 and 2 
are related and are analogous to the rationale 
for the experiment and the literature review 
included in conventional research reports. 
These questions will be addressed in the next 
two sections of this report. Questions 3-6 
involve the collection of baseline data prior to 
implementing the intervention and the collec- 
tion of on-going data during the intervention. 
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Issues pertaining to data collection and analysis 
in a formative experiment will be addressed 
specifically in a subsequent section and as we 
present our methods and findings. As in a 
conventional research report, we conclude with 
a summary and general discussion of our 
findings. 

Our Pedagogical Goal and Its Value 

Our pedagogical goal in this formative 
experiment was to increase the amount and 
diversity of students’ independent reading. In 
addition to being a longstanding, widely ac- 
cepted, and intuitively valued instructional goal 
in reading instruction, research has reinforced 
the centrality of independent reading as a 
powerful means to increase students’ reading 
competency. Theorists and reviewers of exist- 
ing research such as Stanovich (1986) have 
argued that differences in reading ability are 
due in large measure to differences in the 
amount of children’s reading. And, studies 
involving large samples of students nationally 
indicate a strong positive correlation between 
reading out of school and reading achievement 
(see Foertsch, 1992). Similarly, there is evi- 
dence that valued correlates of reading ability 
such as vocabulary knowledge are also related 
to how much children read (see Freebody & 
Anderson, 1983). In addition, independent 
reading is a defining attribute of engaged 
reading, which has been identified as a domi- 
nant theme of the National Reading Research 
Center (see Alvermann & Guthrie, 1993). 
Increasing children’s interest in reading inde- 
pendently has also been identified as having the 
highest priority for research among teachers 



and school administrators (see O’Flahavan et 
al., 1992). 

Despite the substantiated view that the 
amount of students’ independent reading fig- 
ures highly in their reading ability and their 
propensity to read, there is strong evidence that 
many children engage in independent reading 
infrequently, especially outside of school. For 
example, Anderson, Wilson, and Fielding 
(1988) found that reading ranked well behind 
other outside-school activities such as watching 
television and talking on the telephone. A 
comparison of two administrations of the 
National Assessment of Educational Progress 
(NAEP) in 1988 and 1990 (see Foertsch, 1992) 
suggests that there may be a trend toward 
further decreases in the amount of students’ 
independent reading. For example, that com- 
parison found decreases in library use and in 
reading for fun, which was accompanied by an 
increase in the percentage of students reporting 
that they read only fiction or only nonfiction. 
Likewise, approximately half of the students in 
the eighth grade (no data were available for the 
fourth-grade students) reported reading the 
same author either weekly or monthly, which 
perhaps indicates a lack of diversity in their 
reading. 

Thus, the value of setting a pedagogical 
goal to increase the amount and diversity of 
students’ independent reading is justified be- 
cause (a) research indicates that independent 
reading plays a central role in determining 
reading ability; (b) evidence suggests that 
many children do relatively little independent 
reading, especially outside of school, and that 
there is a trend toward further decreases in the 
amount and diversity of independent reading; 
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and (c) surveys of educators identify increasing 
interest in reading as a priority for reading 
research. 

The Instructional Intervention 

The instructional intervention in a forma- 
tive experiment may be one found in the exist- 
ing literature, selected primarily to test its 
relation to a pedagogical theory, or one the 
investigator designs specifically to address the 
pedagogical goal, thus testing an alternative 
means to accomplish the goal (Baumann et al., 
1996). The instructional intervention in the 
present experiment falls within the latter cate- 
gory. That is, the only previous literature 
related to having children create multimedia 
book reviews is our own work carried out 
within a conventional experimental paradigm 
(see Reinking & Pickle, 1993). In that work, 
we report only limited findings related to an 
intervention that used computers but did not 
employ multimedia. In the remainder of this 
section, we elaborate our rationale for design- 
ing the instructional intervention and we pro- 
vide an overview of its components as they 
evolved during the 2 years of implementation 
in three elementary schools. 

Rationale for the Intervention 

Our rationale for selecting multimedia 
book reviews' as an intervention to accomplish 
the goal of increasing the amount and diversity 
of reading was based primarily on its potential 
as an alternative to the conventional required 
book report. In the middle grades, the required 
book report is a ubiquitous instructional activ- 



ity aimed at encouraging students’ independent 
reading. Despite its widespread use, many 
authors have questioned its effectiveness in 
accomplishing that goal . Although no empirical 
evidence can be cited, writers such as Spiegel 
(1981) have argued that the required book 
report may actually subvert teachers’ intentions 
by encouraging some students to read less (by 
selecting shorter books) and more narrowly (by 
avoiding genres or topics that may prove 
unappealing). Likewise, the required book 
report is frequently cited by our undergraduate 
and graduate students as one of the most nega- 
tive memories of elementary school reading 
instruction. In contrast, the opportunity to 
work on a computer has been consistently 
identified as a highly motivating instructional 
activity in schools (Becker, 1990; Reinking & 
Bridwell-Bowles, 1991). 

Although there are many classroom activi- 
ties to encourage independent reading and 
authentic responses to books, there is no evi- 
dence that such activities have replaced the 
wide spread use of conventional book reports. 
We reasoned that using technology to trans- 
form an activity familiar to both students and 
teachers had an advantage over other alterna- 
tives not so firmly entrenched in the dominant 
instructional culture of elementary schools. 
That is, multimedia book reviews are concep- 



'We purposefully used the term book review instead of 
book report when discussing this intervention among 
ourselves and among the teachers and students with 
whom we worked. This decision was based on our desire 
to highlight the differences we perceived between the 
purposes of the conventional book report and the multi- 
media activities with which we wished to engage teachers 
and students. 



NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55 



A Formative Experiment 



9 



tually similar to book reports and therefore are 
not radically different from a familiar school 
activity. This familiarity might enhance the 
potential for integration into existing instruc- 
tional activities, while at the same time the 
unique characteristics of multimedia book 
reviews might counter some of the negative 
aspects of conventional book reports. This 
perspective is consistent with one of the ques- 
tions addressed by formative experiments as an 
approach to classroom research. That is, for- 
mative experiments address whether an inter- 
vention has been fully appropriated by teachers 
and students (see Newman, 1990) or the degree 
to which the educational environment has 
changed as the result of the intervention (see 
question 6 in the framework for designing a 
formative experiment presented in a previous 
section). 

The required book report can also be 
criticized because as typically implemented in 
classrooms it is inconsistent with the meaning- 
ful communicative activities that have been 
shown to enhance students’ reading and writ- 
ing. For example, Kirby and Kirby (1985) in 
their analysis of the research on assigned 
school-related and unassigned out-of-school 
tasks concluded that students who complete 
assigned school tasks take fewer risks in read- 
ing and writing. As typically implemented in 
many classrooms, students write book reports 
primarily for the teacher in order to satisfy a 
requirement or to obtain extra credit. One of 
the advantages of computer technology that has 
been frequently cited in the literature is its 
potential to create opportunities for students to 
engage in meaningful communicative experi- 
ences that frequently involve reading and 



writing (e.g., Bruce & Rubin, 1993; Means et 
al., 1993; Reinking, 1986). There is also some 
evidence that engaging students in creating 
multimedia activities can imbue school reading 
and writing activities with the characteristics of 
out-of-school reading and writing. For exam- 
ple, Turner and Dipinto (1992) concluded in 
their qualitative study of students who became 
hypermedia authors that, 

Their strong sense of audience motivated 
the students to present the information so 
that their peers could understand it 
better. Technology didn’t just enhance 
the appearance of students’ reports, it 
also encouraged them to rethink how to 
present information to communicate it 
more effectively, (p. 198) 



Extrinsic awards are also often associated 
with conventional book report activities. For 
example, a popular restaurant chain provides 
free pizzas for students who can document 
independent reading of books, and many 
schools participate in such programs. Simi- 
larly, a popular commercial computer program 
awards points based on the difficulty and 
length of various books when students can pass 
an on-line test of factual information in the 
books. Teachers will frequently use the points 
as a basis for distributing prizes for reading 
independently. Our pedagogical theory deval- 
ues these approaches in favor of more intrinsi- 
cally motivating activities, a point of view 
substantiated in part by consistent findings that 
tangible rewards simply for completing tasks 
has a somewhat negative effect on intrinsic 
motivation (see Cameron & Pierce, 1994). 
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Figure 2. Comic strip representation of a potential limitation of multimedia book reviews. 
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Thus, there is reason to believe that 
multimedia book reviews, as an alternative 
to conventional required book reports, have 
potential for increasing the amount and 
diversity of students’ independent reading by 
engaging them in personally meaningful 
responses to what they read, by sharing 
those responses through their multimedia 
presentations, and by capitalizing on the 
intrinsic motivation of using a computer to 
share information about what one has been 
reading. 

At the same time, we recognized a 
priori that there may be potential problems 
associated with multimedia book reviews, 
which may undermine their effectiveness in 
accomplishing the pedagogical goal. For 
example, one potential problem is that 
students might have more interest in creating 
multimedia products than in reading books, 
and thus the intervention may actually inter- 
fere with students propensity to read. This 
limitation is reflected poignantly in a recent 
comic strip presented in Figure 2. As this 
concern illustrates, formative experiments, 
unlike conventional experiments, encourage 
researchers to acknowledge that virtually all 
instructional interventions entail some po- 
tentially negative instructional by-products. 
However, formative experiments are designed 
to identify specifically what factors inhibit 
the accomplishment of the pedagogical goal 
and to adapt the intervention or its imple- 
mentation to address those factors that have 
a negative affect on achieving the pedagogi- 
cal goal. 



Overview of the Intervention 
and Computer Programs 

In a formative experiment the intervention 
and the way it is implemented may change as 
parties involved in the research respond to on- 
going data collection aimed at determining the 
factors that enhance or inhibit the accomplish- 
ment of the pedagogical goal. Nonetheless, it 
is possible to describe the intervention in 
general terms, at least as it is initially con- 
ceived prior to the experiment. In the present 
experiment, our initial broad conception of the 
intervention remained intact throughout the 2 
years we worked in three elementary schools; 
although, many details related to its implemen- 
tation evolved and were adapted in response to 
our on-going data collection and analysis. Here 
we provide an overview of the intervention and 
describe the most recent version of the com- 
puter programs in use at the completion of the 
experiment. 

As we have indicated previously in this 
report, we conceptualized multimedia book 
reviews as an alternative to conventional book 
reports. Like a conventional book report, a 
multimedia book review was to be completed 
by students independently after they read a 
book. Unlike a conventional book report, 
multimedia book reviews were to be created 
with the aid of a computer that allowed graph- 
ics and sound to accompany textual informa- 
tion. Another key difference was our intention 
that the multimedia book reviews would be 
compiled into a searchable database available 
for use by students, teachers, parents, and 
other interested individuals. We envisioned 
that this database of students’ multimedia book 
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Figure 3. Template card 1: Main menu screen showing the books a student has read and reviewed. 



reviews would be available in an easily accessi- 
ble location such as a school’s media center. 
As discussed previously, we reasoned that the 
use of the computer to create multimedia 
products had potential to increase students’ 
motivation and engagement when compared to 
conventional book reports. 

Likewise, we reasoned that the database 
was an important extension of the conventional 
book report. The audience for conventional 
book reports is typically the teacher, and book 
reports are usually completed as a required 



school-related task. However, given the pur- 
pose of the database, the audience for multi- 
media book reviews would be much broader, 
and the purpose would be more functional and 
therefore authentic. In addition, the database 
would provide an appealing mechanism for 
students to explore each others’ reading and to 
find books they might like to read. Similarly, 
although each multimedia book review would 
be an individual student’s product, we envi- 
sioned much collaborative work among stu- 
dents as they assisted each other in dealing 
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TEACHERS 



About 

Me! 



My 

Books 



More about 
this book 



Figure 4. Template card 2: Review screen completed by students for each book reviewed. 



with the new technological skills needed to 
create multimedia book reviews. Taken to- 
gether, we believed that these differences 
between conventional book reports and multi- 
media book reviews had potential to transform 
a common school activity in a way that would 
have a positive affect on the amount and diver- 
sity of their independent reading. 

To provide students and teachers with 
much flexibility in creating multimedia book 
reviews and to increase their involvement in 
this alternative (as well as its potential to be 



appropriated fully into the instructional envi- 
ronment), we decided to teach teachers and 
students how to use HyperCard (Version 2.1). 
HyperCard is an authoring system for the 
Macintosh computer, and it can be used to 
create multimedia presentations. It is a power- 
ful, open-ended application for creating com- 
puter programs, but it permits a user to create 
relatively sophisticated computer applications 
with only basic knowledge of its operation. We 
planned to acquaint teachers and students 
with the rudimentary tools necessary to create 
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Figure 5. A student’s drawing (using clip art and 
drawing tools) from the “More about this book” 
option on the review card (see Figure 4). 



HyperCard programs referred to as “stacks,” 
which is an extension of the HyperCard meta- 
phor of screens as “cards” that are linked by 
clicking on “buttons.” We taught them how to 
use all of the basic HyperCard tools short of 
scripting, which is a more advanced level of 
HyperCard programming. 

One dilemma that we faced early in our 
planning and implementation of the interven- 
tion was how to design the multimedia book 
reviews to be open-ended and personal while 
accommodating the standardization necessary 
for the database to be searched systematically. 



In other words, the database of multimedia 
book reviews that we envisioned required that 
information be embedded into established 
categories. However, if we required students to 
enter information about the books they read 
into predetermined categories, we might 
subvert their creativity and their personal 
responses to creating each book review. 

Our successive attempts during the experi- 
ment to resolve this dilemma and to notice 
what effects it had on students and teachers is 
one example of why this approach to research 
is described as formative. That is, the solution 
to our dilemma emerged through a series of 
adaptations and compromises in response to 
what we observed during the experiment. 
Some of these adaptations include: (1) seeking 
teachers’ and students’ input into what catego- 
ries of information about the books they read 
that they thought most relevant; (2) seeking 
their input about the layout of these categories 
on a standard card used by all students, which 
came to be referred to as the book review 
“template”; and (3) creating nonsearchable but 
personalized sections of the template (e.g., 
boxes for pictures and photographs, an audio 
button that played a student’s comments about 
the book, and a card that provided an autobio- 
graphical sketch of each student reviewer). In 
addition, we eventually created a button that 
led to a separate nonsearchable stack that 
students could choose to create as their per- 
sonal supplement to the basic template. 

The most recent book review template, 
which is similar in many respects to earlier 
versions, is comprised of three screens 
(“cards” in HyperCard) presented in Figures 3, 
4, and 6. The screen shown in Figure 3 is a 
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Figure 6. Template card 3: Student-written biographical sketch. 



template card 1 that serves as a main menu 
comprised of the student’s picture and 12 icons 
representing the books that a student has read 
(additional screens can be added when a student 
has entered more than 12 books). A student’s 
multimedia review of a particular book can be 
accessed by clicking on the appropriate book 
icon. Other components of the program can be 
accessed by clicking one of the buttons on the 
memo bar at the bottom of the screen. This 
menu bar appears at the bottom of all template 
cards and serves as the primary means for 



navigating among the book reviews and the 
database functions. The addition of this menu 
bar to earlier versions of the program illus- 
trates the formative aspects of creating a com- 
puter application that operated efficiently 
within the larger goals of the experiment. 

Template card 2, which serves as the main 
review screen, is shown in Figure 4. In addition 
to the menu bar, it is comprised of (a) textual 
fields into which students can enter descriptive 
information about the book as well as a sum- 
mary and their comments about the book, (b) an 
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[Clicking the 'Fetch Fido” button initiates a search through 
all of the students' book revievs for books authored by E. 
B. White, as specified in the search criteria above. The 
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listing vould include title, author, and reviever for any 
reviev meeting the search criteria. A user can then click 
on any book listed to see hov a student revieved itj.] 
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Figure 7. “Fido Fetch” screen used to direct the search of the data base containing all students’ reviews. 



audio button that plays students’ comments 
or sound effects, and (c) and navigation 
buttons, including one (“more about this 
book”) that allows students to add additional 
cards to their own specifications. For exam- 
ple, on these additional cards, many students 
chose to include pictures from clip art files, 
often modified or supplemented by their own 
art work created on-line with the drawing 
tools, and additional information about the 



book’s authors or characters (see Figure 5). 
The textual fields are scrolling fields so that 
a student has unlimited space to enter text 
despite the small area displayed initially on 
this screen. All of the textual fields can be 
searched when using the database application 
described momentarily. 

Figure 6 shows the third template card, 
which is the screen that students used to 
enter biographical information. Their bio- 
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Figure 8. Screen showing teachers whose students’ reviews can be accessed by clicking on a teacher’s 
picture. 



graphical sketches can be accessed by click- 
ing the button on the menu bar labeled 
“About Me!” 

The database application, which we 
called “Fido Fetch,” was operated from the 
card shown in Figure 7. To search the data- 
base, a user would select one of the catego- 
ries to be searched, each category corre- 
sponding to the textual fields on the main 
review template (see Figure 4). After enter- 



ing a key word or phrase in the textual 
field on the upper part of the screen and 
clicking on the fido fetch button, the data- 
base application would find all reviews 
containing the key word or phrase speci- 
fied. These reviews would be displayed by 
title, author, and reviewer in the scrolling 
textual field on the lower part of the 
screen. Clicking on one of these reviews 
would display the full review. 
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Figure 9. Screen showing students in a class whose reviews can be viewed by clicking on a student’s 
name. 



Initial entry into the reviews and the 
database application was achieved from the 
cards shown in Figures 8 and 9. The 
screen in Figure 8 displays pictures of 
teachers whose students have entered book 
reviews. A user could click on a teacher to 
access a list of students in that teacher’s 
class (Figure 9) and then access a particular 
student’s main book review menu (Figure 3) 
by clicking on the student’s name in the list. 



Method 

Research Sites 

This formative experiment was conducted 
in two elementary schools during the 1992-93 
school year and in one elementary school dur- 
ing the 1993-94 school year. The schools were 
selected from among six schools contacted that 
had expressed an interest in the project and 
that had or were willing to purchase the mini- 
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mal hardware needed to implement the inter- 
vention. The final selection of research sites 
was made by the university researchers based 
on the schools’ location, equipment and facili- 
ties, enthusiasm for and commitment to the 
project, student populations, and so forth. Of 
the two schools that participated during the 
1992-93 school year, one school (hereafter 
referred to with the pseudonym “Collins 
School”) was located in a small town within 
commuting distance of a large metropolitan 
area. The school was relatively large, accom- 
modating 3-4 classrooms of approximately 30 
students at each grade level K-5. Students and 
teachers reflected the community’s homoge- 
nous population consisting of predominantly 
White, middle- and upper-middle-class fami- 
lies, many of whom commuted to work in the 
nearby large metropolitan area. We worked 
with two fourth-grade classes and their teachers 
at Collins School . The second school (hereafter 
referred to with the pseudonym “Hartwig 
School”) was located in a rural area, although 
its students came from blue-collar and pro- 
fessional homes in a small town as well as 
from agricultural areas. Approximately 15% 
of the Hartwig student population was Afri- 
can American. The school was moderately 
large, accommodating approximately 2-3 
classrooms of 25-30 students at each grade 
level (K-5). We worked with 3 fifth-grade 
classes, their respective teachers, a Chapter 1 
teacher, and a gifted teacher in Hartwig 
School. The third school (hereafter referred 
to with the pseudonym “Borders School”) 
was in the same school district as Hartwig 
School and had a similar, although some- 
what smaller, student population. At Borders 



School, we worked with 1 fifth-grade and 1 
fourth-grade class, their respective teachers, 
and a Chapter 1 teacher who also managed the 
computer lab. 

Hartwig and Borders schools each had a 
small computer lab with 12 and 10 Macintosh 
LC II computers, respectively. Both schools 
had access to a color scanner for digitizing 
pictures and several printers. Borders school 
also had a black and white LCD panel for 
displaying Macintosh output on a standard 
overhead projector. In comparing the imple- 
mentation of the intervention at the three 
schools, we found that the LCD panel was an 
invaluable tool in teaching students Hyper- 
Card. It greatly streamlined that phase of 
implementing the intervention. Collins school 
had a computer lab of Apple He computers, but 
purchased two Macintosh LC II computers and 
a scanner to be shared by two classes involved 
in the experiment. During the first year of the 
project, we were interested in discovering the 
logistical and pedagogical implications of 
implementing the intervention in a lab setting 
where students worked for approximately 1-2 
hr per week in the computer lab when com- 
pared to a setting in which a single computer 
was available constantly in a classroom. Based 
on our experiences, these two alternatives 
represent typical patterns of availability in 
elementary schools (see also Becker, 1990, 
1992a, 1992b). Likewise, our goal at Collins 
School was to minimize the amount of knowl- 
edge students would need about HyperCard in 
order to enter their book reviews, relying 
instead on training teachers and parent volun- 
teers to assist with HyperCard applications 
developed by the university researchers. 
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Participants 

The composition of the university research 
team varied during the course of the project, 
but included the principal investigator (the first 
author of this report), three consultants, and 
five doctoral students who provided assistance 
at various stages of the project in areas such as 
solving technological difficulties, collecting 
and analyzing data, teaching HyperCard to 
teachers and students, and writing the final 
report. The three consultants included Thomas 
Reeves, a Professor of Instructional Tech- 
nology at the University of Georgia (UGA); 
Mary Jo Brown, an Assistant Professor at 
UGA who teaches courses in qualitative re- 
search; and Valerie Garfield, a middle-grades 
teacher who has had much successful experi- 
ence in integrating computers into her teaching. 
Although we are indebted to them for their 
input and assistance, this report reflects only 
our own summary and interpretations. 

Consistent with the goals of the National 
Reading Research Center (see Alvermann & 
Guthrie, 1993) and the nature of formative 
experiments, we considered the classroom 
teachers with whom we worked on this project 
to be integral members of our research team. 
We continually sought their interpretations of 
what was occurring in their classrooms relative 
to the experiment, and we continually solicited 
their suggestions for modifying the interven- 
tion. Members of the university research team 
met regularly with the teachers to discuss their 
observations of what was occurring and to plan 
future strategies and modifications. Teachers 
also kept written logs of their observations, 
which were discussed periodically with a 



university researcher. In addition, teachers 
with whom we worked were co-authors on 
papers presented at two national professional 
meetings and the teachers participating in the 
project at Borders school appeared without the 
university researchers on the program of a state 
professional meeting where they made a pre- 
sentation related to their involvement in the 
project. 

Nonetheless, it may not be accurate to 
consider the present formative experiment as 
an example of collaborative research given 
current definitions (cf. Allen, Buchanan, 
Edelsky, & Norton, 1992; Anders, 1996; 
Jervis, Carr, Lockhart, & Rogers, 1996). We 
did not involve the teachers in selecting the 
pedagogical goal (although they clearly agreed 
that it was an important one), nor did we 
formally involve them in analyzing data. Nei- 
ther did we involve them directly in preparing 
this report. Also, we observed and analyzed 
teachers’ reactions and behaviors during the 
time we were working in the schools as part of 
the experiment without systematically sharing 
our conclusions with them until after the exper- 
iment. Thus, in this report we use pseudonyms 
to refer to the teachers with whom we worked 
closely. 

We wish to note that our decision to not 
involve teachers fully as equal partners in the 
research process was not based on our dis- 
agreement with the more stringent standards 
outlined in the emerging literature on collabo- 
rative research. Our stance in this project was 
consistent with that literature, which suggests 
that truly collaborative research in which 
teachers define themselves as researchers tends 
to evolve over an extended time and across a 
number of collaborative experiences . Given the 
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separate professional cultures of researchers 
and teachers that is prevalent today, it may not 
be possible to generate the trust, confidence, 
and understanding necessary for a true collabo- 
ration without an extended period of interaction 
(see Anders, 1996; Jervis et al., 1996). As we 
write this report, further collaborations be- 
tween some of the teachers and the university 
researchers are being planned. We are confi- 
dent that these future collaborations will move 
increasingly towards the ideal of truly collabo- 
rative research and that the teachers involved 
will increasingly define themselves as research- 
ers. 

In general, the participants in this experi- 
ment did not deviate in significant ways from 
the researchers, teachers, and students one 
would expect to find in their respective envi- 
ronments. The researchers from the university 
had the orientations and values one would 
expect of other doctoral students and faculty in 
a major research-oriented university, albeit we 
believe with a heightened awareness of collabo- 
rative research and of the value of teachers as 
professional colleagues. From the outset of this 
project, we also were excited about the poten- 
tial of formative experiments as a new ap- 
proach to classroom intervention research, and 
we were eager to explore the new insights it 
might produce about computer technology and 
literacy and about our own understandings of 
research. 

In general, the schools and classrooms in 
this investigation reflect the organization and 
operation typical of many elementary schools 
serving the community populations described 
previously. Classrooms were self-contained for 
most of the day, although students did occa- 



sionally receive instruction from other teachers 
who taught subjects such as Art, Physical 
Education, and Band, and these times often 
provided an opportunity for teachers to have a 
planning period. A relatively small percentage 
of the students in each classroom left periodi- 
cally for special instruction in the Chapter 1 or 
gifted programs. A commercial basal reading 
series figured prominently in reading instruc- 
tion. Although two of the teachers in Collins 
school described themselves as having a whole 
language orientation, we did not observe their 
practices to deviate greatly from the other 
teachers who used basals extensively. Prior to 
their involvement in this experiment, all of the 
teachers devoted some time to classroom 
activities aimed at encouraging independent 
reading. Such activities included requiring 
students to write conventional book reports, 
reading sections of books that they thought 
students might want to read, and putting up 
bulletin boards showing the number of books 
read by each student in the class. For the most 
part, the teachers made available books for 
independent reading through a classroom 
library rather than through regular trips to the 
school library. 

As determined through a structured inter- 
view (see Appendix A), all of the teachers had 
some background with computers in their 
teaching, although the amount and type of that 
experience varied somewhat. Through observ- 
ing their classrooms and daily routines during 
a baseline period before the multimedia book 
review activity was introduced, we found that 
all of the teachers were actively using the 
computer in conjunction with their teaching 
activities, but they did so in ways that were 
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Figure 10. Phases of the experiment and approximate time line of events. 



largely perfunctory and not fully integrated into 
their instructional program. For example, the 
following comment by one of the teachers was 
typical. 

I use it [the computer] mostly for when 
they finish their work and enrichment 
type [activities]. The games that were on 
[the computer] were just to reinforce 
skills. But not everyone had access to it 
because there were always the ones who 
never go to it. 

The teachers were all enthusiastic about discov- 
ering more ways to use the computer in their 
instruction, which was one of our criterion for 
choosing to work with them in this project; 
however, they all acknowledged some degree 
of trepidation about technology, which they felt 
was typical of but not as extreme as their peers. 
For example, in an interview one teacher 
stated, 

We had a workshop this summer on 
using the computer as an instructional 
tool. For anyone who wanted to sign up. 

. . . They did a survey, and there was a 



big majority that was terrified. They had 
no experience, they didn’t even want to 
touch it. But after the workshop, our 
staff has [begun] to feel like its not going 
to bite if they touch it. 

Beyond these similarities, there were clearly 
important individual differences among teach- 
ers, students, and classrooms relevant for this 
study, but these differences will be highlighted 
subsequently in our presentation of findings 
and interpretations. 

Procedures 

The research project proceeded through 
several phases during the first year in Collins 
and Hartwig Schools, and these phases were 
repeated with minor variations during the 
second year in Borders School. The phases of 
the project and the events associated with each 
phase are described in this section and are 
summarized in Figure 10. 

Phase 1 . Phase 1 took place in the spring 
prior to the school year during which the 
intervention was to be implemented. At that 
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time, representatives of the university research 
team met with teachers, the school principal, 
and in some cases district office personnel to 
discuss the project, what expectations there 
would be for the various participants, needed 
hardware, and so forth. 

Phase 2. When expectations were clearly 
understood and supported by all participants, 
we proceeded to Phase 2, which took place 
during the summer. During several summer 
meetings, members of the university research 
team trained teachers and in some instances 
parent volunteers and teacher aids to use the 
Macintosh LC II computer as well as the basic 
tools and operations of HyperCard. We also 
discussed with the teachers our conceptions of 
the multimedia book reviews, how we thought 
they might be implemented, the nature of 
formative experiments, logistical problems, 
data collection, and so forth. Together with the 
teachers, we developed an initial plan and 
schedule for implementing the multimedia book 
review project beginning early in the fall. 

Phase 3. Phase 3 occurred during approx- 
imately the first 6 weeks of the school year. 
During that time, members of the university 
research team gathered baseline data including 
qualitative observational and interview data 
aimed at creating a rich description of the 
school and classroom environment as well as of 
the teachers and students. We also gathered 
quantitative data aimed specifically at determin- 
ing the amount and diversity of students’ inde- 
pendent reading and their attitudes toward 
reading. Toward that end, teachers adminis- 
tered the Elementary Reading Attitude Survey 
(McKenna & Kear, 1990), a standardized 
instrument designed to measure students’ 



attitude toward reading in school and out of 
school. Teachers also administered a student 
questionnaire designed by the university 
research team to determine particularly the 
diversity of students’ reading (see Appendix 
B). A parent questionnaire was also designed 
by the university researchers to determine 
parents’ perceptions of students’ independent 
reading outside of school (see Appendix C). 
This questionnaire was sent home with a 
letter signed by the principal investigator 
and the classroom teacher explaining the 
school activities that would be taking place 
in conjunction with the project and encour- 
aging parents to communicate questions, 
concerns, and comments to us. The principal 
investigator also attended a fall meeting of the 
Parent Teacher Organization in each partici- 
pating school to explain the project and 
answer questions. Also during Phase 3, 
teachers completed the DeFord Teacher 
Orientation to Reading Profile (TORP; see 
Deford, 1985) so that we might obtain infor- 
mation about teachers’ views of reading and 
the teaching of reading. 

All of these quantitative and qualitative 
sources of baseline data were also employed in 
three comparison-classroom classrooms in 
Collins School. These classrooms were se- 
lected because they represented similar popula- 
tions of students and because the teachers in 
these classrooms were using an alternative, 
computer-based approach to enhancing stu- 
dents’ independent reading. This approach was 
centered in the use of a commercial software 
program that automatically tested students’ 
factual knowledge of a book they had read and 
awarded points based on their performance. 
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Teachers rewarded students in various ways 
based on the number of points they received. 

One development during Phase 3 at Bor- 
ders School is noteworthy because it clearly 
illustrates the differences between formative 
and conventional experiments. Approximately 
1 month into the school year, one of the teach- 
ers participating in the project left the school to 
take another position. Her replacement knew 
little of the project when she arrived at the 
school to take over the class. This development 
would be seen as a severe threat to internal 
validity in a conventional experiment. How- 
ever, from the perspective of a formative 
experiment, we saw this as a new opportunity 
to learn about how the project activities could 
accommodate the unforeseen variations that are 
characteristic of dynamic educational contexts 
and that can affect instructional interventions in 
schools. 

Phase 4. In Phase 4, students were taught 
how to program in HyperCard up to the level 
of scripting. They were informed that the 
ultimate purpose for learning HyperCard was 
so that they could create multimedia book 
reviews. That is, they learned how to use all of 
the HyperCard tools for drawing, copying and 
pasting graphics, creating buttons and text 
fields, linking cards, and so forth. Phase 4 was 
carried out in Hartwig and Borders Schools but 
not in Collins School where a computer lab 
was unavailable to teach HyperCard. There 
were important differences between Phase 4 in 
Hartwig School during Year 1 and in Borders 
School during Year 2, and these differences 
reflect the formative aspects of this approach to 
research. In Hartwig School, students came to 
the computer lab once a week for instruction in 



HyperCard where they worked individually or 
in pairs at a computer. The instruction was 
provided by a member of the university re- 
search team, particularly one graduate student. 
The teachers brought their students to the lab 
and typically assisted students in carrying out 
HyperCard tasks as directed by the graduate 
student. Other members of the university 
research team along with a Chapter 1 teacher 
and parent volunteers who had been trained in 
HyperCard were also typically available to 
assist students. 

The weekly lessons and activities for 
students were typically designed and modified 
from week to week based on an analysis of the 
previous week’s lesson and observations of 
students’ reactions and understandings as we 
proceeded to teach them HyperCard. The 
weekly lessons were also specifically designed 
to prepare students to create multimedia book 
reviews (e.g., examples and sample activities 
related to information about books). Between 
the weekly lessons, the Chapter 1 teacher 
worked with students who had been absent or 
who needed additional help. Phase 4 required 
approximately 18 weeks in Hartwig School 
during the first year of the project. By the end 
of Phase 4, all students were minimally compe- 
tent in using HyperCard, as indicated by an 
informal assessment task requiring students to 
use all of the HyperCard skills they had 
learned. During all of the weekly lessons, 
observational data were recorded by a member 
of the university research team. 

Based on our experiences and analysis of 
data during Year 1 at Hartwig School, the 
university research team developed a series of 
lesson plans aimed at teaching students how to 
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program in HyperCard. 2 These lessons were 
presented to the teachers involved with the 
project at Borders School during two 4-hr 
sessions during the summer (Phase 2) before 
the start of Year 2. Several of the teachers 
from Hartwig also participated in these ses- 
sions. Our intent was to discover whether 
teachers with relatively little background in 
using computers and who had no knowledge of 
HyperCard could be trained in a reasonable 
time to teach their students HyperCard with the 
lesson plans, materials, and activities we had 
developed. At Borders School, the Chapter 1 
teacher presented the content of the lessons to 
whole classes for 1 hr a week in the school’s 
media center using an LCD panel. She fol- 
lowed up this whole-group presentation on 
Monday with groups of 8-12 students on 
Tuesday through Friday. In the small groups, 
students reviewed the content of the lesson and 
completed activities on computers in the lab for 
approximately 40 min each. Under these condi- 
tions, students were able to master HyperCard 
programming in approximately 10 weeks. 
During Phase 4 at Borders School, teachers 
critiqued the effectiveness of the lesson plans in 
writing and made suggestions for further im- 
provements. Members of the university re- 
search team were also collecting and analyzing 
observational, interview, and video data during 
that time. 

Phase 5. In Phase 5, the book review 
template was finalized and students entered 
information about the books they were reading 



^he lesson plans and related materials are available as an 
instructional resource from the NRRC (Reinking & 
Bonham, 1996). 



into the template. However, the template and 
its use varied across the three schools. In 
Collins School, the university research team 
created the template based on the specifications 
decided by students and teachers, because 
students were not taught how to program in 
HyperCard. Thus, students at Collins school 
were limited to the categories of information 
available on the template. At Hartwig School, 
students not only created the template them- 
selves (with some guidance from teachers and 
the university researchers), but they also were 
able to use their HyperCard skills to go beyond 
the standard review template. We found that 
this approach had advantages but also pre- 
sented problems. For example, when students 
created their own version of the standard 
template, there were frequently small varia- 
tions, omissions, or programming errors in 
their HyperCard stacks that made the possibil- 
ity of a database search problematic. At Bor- 
ders School, we compromised these extremes 
by developing a standard working template for 
students while adding a button to the template 
that would enable them to use their HyperCard 
skills to enter personalized information about a 
book they had read. 

Phase 6. Phase 6 involved compiling 
students’ book reviews into a database and 
making the database available for use in a 
central location. A number of factors prevented 
us from reaching this phase during the first 
year in Collins and Hartwig Schools. Many of 
the same factors explained why we did not 
reach Phase 6 until the end of April in Borders 
School during the second year of the project. 
Mainly, we had underestimated the interact- 
ing logistical, pedagogical, and technological 
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problems associated with teaching teachers and 
students HyperCard, creating workable pro- 
grams, and consolidating the book reviews into 
the database so that they could easily be 
searched. Although the purpose of a formative 
approach is to discover necessary adaptations 
and adjustments, we underestimated the num- 
ber and extent of these revisions. Thus, a 
major component of the intervention aimed at 
addressing the pedagogical goal was imple- 
mented only marginally during the second year 
of the project. However, we have data indicat- 
ing that creating multimedia book reviews, 
even if they are not entered and searched into 
a database, effects changes relevant to our 
pedagogical goal. 

Phase 7. In Phase 7, we had students, 
teachers, and parents complete again the writ- 
ten surveys administered in Phase 3. These 
data were gathered during the last few weeks of 
the school year in May and allowed us to 
compare any differences that may have oc- 
curred in responses on these instruments since 
the baseline data were collected at the begin- 
ning of the school year. Data were collected in 
the classrooms using the multimedia book 
review intervention as well as the three com- 
parison classrooms using the computer-based 
point system. 

Data Collection and Analysis 

In this section, we present the types of 
data we gathered and how we gathered it. We 
also outline our perspectives on how we view 
data collection within our emerging under- 
standing of a formative experiment. Our origi- 
nal and current perspectives provide a context 



for reporting our results in the following sec- 
tion. 

Quantitative data were gathered during 
Phases 3 and 7 of the project, as described in 
the previous sections. Although we chose to 
collect these data, we do not believe that 
quantitative data are essential to conducting 
a formative experiment. We see formative 
experiments existing within the domain of 
research that Salomon (1991) has described as 
systemic, a descriptor that he argues transcends 
qualitative and quantitative paradigms (see also 
Eisenhart & Borko, 1993). Neither is a pre-/ 
post-intervention comparison, as we have 
done, essential. The goal of formative experi- 
ments to modify instructional interventions to 
accomplish a pedagogical goal necessitates that 
they be more open ended than conventional 
experiments. Newman (1990) provides a basis 
for this open-endedness from a sociohistorical 
perspective of cognitive change that draws 
heavily on the Vygotskian concept of the zone 
of proximal development (ZPD; see also New- 
man, Griffin, & Cole, 1989). He sees com- 
puter technology as a means for bringing 
teachers’ and students’ interactions into the 
ZPD because it amplifies teachers’ capability 
to organize the educational environment. Yet, 
this sociohistorical perspective leads him to 
reject the pedagogical goal as a static endpoint 
for all students. Many different endpoints are 
possible, some of which may transcend the 
original goals. Thus, the goal identified in a 
formative experiment implies no specific 
standard for measuring achievement of the 
pedagogical goal, and the goal can never be 
achieved in an absolute sense. A formative 
experiment is not completed when a predeter- 
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mined criterion has been reached but instead 
stops at some point because of practical con- 
straints (e.g., a school year ends). 

Nonetheless, formative experiments must 
provide a substantiated baseline description of 
students and the educational environment 
relative to the pedagogical goal as a reference 
point from which progress can be gauged. Data 
supporting the description may be quantitative, 
qualitative, or both. Beyond baseline data, a 
formative experiment requires on-going, con- 
tinuous data collection aimed at determining 
what factors enhance or inhibit the interven- 
tion’s success in accomplishing the pedagogical 
goal, what the unanticipated by-products of the 
intervention are, and how the degree to which 
the intervention is being appropriated into the 
educational context. 

Thus, in addition to the quantitative data 
gathered in Phases 3 and 7, qualitative data 
were gathered during Phases 2-6. Sources of 
qualitative data included (a) taped, semistruc- 
tured interviews with teachers (e.g. , about their 
experiences with computers, see Appendix A); 

(b) log books in which teachers recorded their 
observations about events related to the project; 

(c) focus-group discussions with students; (d) 
observational field notes; and (e) videotapes of 
various events during the project. In addition, 
we examined students’ book reviews on the 
computer, both qualitatively (e.g., to examine 
the types of books they were reading) as well 
as quantitatively (e.g., the number of books 
they were entering). These data were analyzed 
for the purpose of understanding what effects 
the intervention was having on the pedagogical 
goal and on other related aspects of the educa- 
tional environment. 



From the outset of this investigation, we 
had a conceptual understanding that formative 
experiments entailed collecting data continu- 
ously to guide iterative modifications of the 
instructional intervention in order to enhance 
its effectiveness in accomplishing the pedagogi- 
cal goal. However, the absence of detailed 
models of formative experiments in the litera- 
ture prevented us from knowing exactly how 
this might be accomplished in practice. Origi- 
nally, we imagined that data collection and 
modifications to the intervention would pro- 
ceed through well-defined cycles. In practice, 
we found that such distinct cycles did not 
occur. Instead, the relation between data col- 
lection and modification of the intervention 
tended to be fluid, even at times ad hoc in the 
sense that adaptations were based on the intu- 
itive demands of the moment rather than a 
rigorous analysis of data. 

In our reflection on this investigation as 
an example of a formative experiment, we 
have considered several explanations for this 
relation. First, we discovered early during 
the first year of the project that we had been 
too ambitious in deciding to work in two 
schools simultaneously. Likewise, we under- 
estimated the complexity of implementing 
the intervention in two schools where the 
technological infrastructure necessary to 
support the intervention was not firmly in 
place. Consequently, during the first year, 
we were not able to engage in the level of 
systematic data collection and detached 
reflection and analysis that we had hoped to 
achieve. Neither were we able to implement 
the database activity during Year 1 and only 
partially in Year 2. 
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Another explanation was our discovery 
that our research activities during the experi- 
ment were formative in several different 
areas simultaneously. For example, we 
continuously made adjustments that were 
responses to factors in the following areas: 
logistical (e.g., How could we create more 
time in the daily schedule for students to 
work on the computer), methodological 
(e.g., Our observations of students working 
in the lab on one day might suggest that we 
needed to interview the teachers about some- 
thing we observed on the following day), 
pedagogical (e.g., How might we adjust 
implementation to encourage more poor 
readers to create book reviews?), technologi- 
cal (e.g., The search speed on the database 
program is too slow. How 'can we make it 
run faster?), interpersonal (e.g., How can 
we as university researchers establish good 
rapport with the teachers; maintain teacher 
morale; take into account the needs of teach- 
ers, administrators, parents, and support 
personnel; etc.), and ethical (e.g., Should 
we adjust the way we implement the inter- 
vention based on our values or the teachers’ 
values?). In addition, adjustments made in 
response to a concern in one area would fre- 
quently have implications requiring adjustments 
in another area. The interacting effects of 
adjustments in multiple areas required that we 
be less structured than we anticipated in our 
approach to conducting a formative experi- 
ment. However, at the same time this real- 
ization reinforced our belief that formative 
experiments deal realistically with the inher- 
ent complexities of implementing instructional 
interventions and researching their effects. 



Taking a formative stance, we became 
aware of the unanticipated challenges and 
complexities we were encountering; we made 
several adjustments that affected data collection 
and analysis. For example, we decided to work 
in one school during the second year of the 
project, where we would also have the added 
benefit of our first-year experiences. 

In addition, we decided to gather more 
in-depth data on 4 focal students in each 
class. These 4 focal students were identified 
by the teacher as being representative of 
students in each of the following categories 
based on combinations of reading ability and 
interest in reading independently: (a) above 
average in ability and in interest, (b) above 
average in ability but below average in 
interest, (c) below average in ability but 
above average in interest, and (d) below 
average in ability and interest. To insure the 
validity of selecting students to represent 
these categories, a member of the university 
research team who had collected baseline 
data in the classroom and the students’ 
teachers from the previous school year also 
independently categorized students along 
these dimensions before being aware of the 
teachers’ selections. There was no disagree- 
ment that each focal student fell within the 
categories identified. Teachers and members 
of the research team took steps to insure that 
focal students were not aware that they had 
been selected for special attention during 
data collection. For example, we observed 
and recorded our observations about focal 
students inconspicuously, and we noted their 
responses to specific questions that were 
addressed to a group of students or, when 
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addressed to them individually, were done so 
informally as opposed to a formal interview. 

Our formative stance toward data collec- 
tion, which led to on-going adjustments in what 
data we gathered and how we analyzed it, is 
consistent with current thinking in qualitative 
and ethnographic approaches to research (see 
LaCompte & Preissle, 1993). As is accepted 
practice in qualitative studies, the data deemed 
most useful emerged from our on-going analy- 
sis and emerging interpretive theories (Glaser 
& Strauss, 1967), but also from our evolving 
understanding of formative experiments. The 
concept of a formative experiment provided a 
broad theoretical framework for collecting and 
analyzing data. However, currently, formative 
experiments are methodologically and theoreti- 
cally ill-defined. At the same time, the general 
concept of a formative experiment implies 
adaptability in collecting data (Baumann et al., 
1996). Thus, theoretical frameworks for data 
collection and analysis were varied in response 
to on-going developments within the project, 
our emerging theories based on data collected 
earlier in the project, and our evolving under- 
standing of formative experiments. 

For example, we began gathering obser- 
vational data in classrooms using Glaser and 
Strauss’ (1967) “frame work of local concepts” 
as an approach to determining the amount and 
diversity of students’ independent reading, 
which was critical to gauging progress toward 
the pedagogical goal of this formative experi- 
ment. Our local framework consisted of our 
experiential knowledge of classrooms, teach- 
ers, and students in general. This knowledge 
suggested that observation of the regular class- 
rooms and the computer lab would be a logical 



initial choice as an approach to data collection. 
As we transcribed and annotated field notes, 
we determined that this approach created too 
many gaps in our understanding of students’ 
independent reading. Thus, we expanded our 
data collection and analysis to include focused 
group discussions with teachers and students 
using techniques and theoretical perspectives 
described in Morgan (1993). 

Another example illustrates how our 
evolving understanding of formative experi- 
ments affected our selection of theoretical 
frameworks for data collection and analysis. 
As stated previously in this section, we discov- 
ered during the project that a formative stance 
toward the instructional intervention applies to 
several distinct but interacting areas. We real- 
ized that a formative approach to classroom 
research and the emergence of these distinct 
areas were consistent with Patton’s (1990) 
description of using qualitative research meth- 
ods, specifically a process/outcomes matrix, to 
evaluate programs: 

The linkage between processes and out- 
comes is a fundamental issue in many 
program evaluations. An evaluation 
research design based on qualitative 
methods can be particularly appropriate 
where either program processes or 
program impacts, or both, are largely 
unspecified, for whatever reasons. Some- 
times the reason is because outcomes are 
meant to be individualized; sometimes 
the program is simply uncertain about 
what the outcomes will be; and in many 
programs neither processes nor impacts 
have been carefully articulated. Under 
such conditions, one purpose of the 
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evaluation may be to help articulate 
program processes, program impacts, 
and the linkages between the two. This 
task can be facilitated by constructing a 
process/outcomes matrix to organize the 
data. (p. 415) 

Although we did not set out to create a 
process/outcomes matrix, our concept of a 
formative experiment guided us to look for 
linkages between processes and outcomes 
under the conditions that Patton describes. 
Interestingly, our efforts led us to formulate six 
distinct areas that could become the foundation 
of such a matrix (i.e., logistical, methodologi- 
cal, pedagogical, technological, interpersonal, 
and ethical). Thus, we conclude that Patton’s 
theoretical framework for program evaluation 
may be a useful one for conducting formative 
experiments. 

Results 

As with any approach to research involv- 
ing inductive methods over an extended period 
of time, it is not feasible to report in detail all 
of the data collected in this formative experi- 
ment. Of necessity, the data reported represent 
the researchers’ synthesis and interpretation of 
events, and it may be organized and presented 
in various ways. We have chosen to use a 
format that Patton (1990) refers to as “key 
events,” which he defines as critical incidents 
or major events, not necessarily presented in 
their order of occurrence. To present our 
results, we chose events that illustrate our 
major findings or that were pivotal in determin- 
ing how the intervention was implemented to 



accomplish our pedagogical goal. The key 
events are organized into themes indicated by 
the headings in this section. Despite our real- 
ization that the formative adjustments during 
the experiment occurred in response to factors 
in a variety of areas as we discussed previ- 
ously, we report here only those adjustments 
related to these key events. 

In the discussion section, we will summa- 
rize and discuss the results within the frame- 
work of the questions guiding a formative 
experiment as presented earlier in this report. 
Before presenting key events, we provide 
baseline data concerning the amount and diver- 
sity of students’ independent reading as a 
reference point for considering progress toward 
this experiment’s pedagogical goal. 

Baseline Data 

Baseline data were gathered during Phase 
3 of the experiment, which extended from 
about the second week of the school year and 
continued through mid-October. Data collec- 
tion during this period focused on (a) deter- 
mining the extent and diversity of students’ 
independent reading and on (b) generating a 
general description of the school and classroom 
environments, including teachers’ orientations 
toward instruction and the use of computers. 
Sources of data included the quantitative instru- 
ments described in the procedures section of 
this report, observational data, and focused 
group interviews with students and individual 
interviews with teachers. Because we have 
already briefly described the school and class- 
room contexts and will continue to explicate 
them in conjunction with presenting our results 
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in this report, we will focus here on data 
related to the amount and diversity of inde- 
pendent reading and to teachers’ orientation 
toward reading. 

Through our baseline observations and 
focused group interviews with students at each 
school, we determined that there was already a 
good deal of independent reading occurring in 
the classrooms with which we worked in this 
investigation. We frequently observed students 
reading during the school day; many had books 
in their desks; they knew where the local 
library was; they read at least occasionally at 
home, and so forth. Our observational data 
indicated that although there were a few stu- 
dents in every class who did very little, if any, 
independent reading, the majority of students at 
least occasionally engaged in independent 
reading. The variations between classrooms 
and across schools in this regard was surpris- 
ingly small. Similarly, the mean raw scores on 
Elementary Reading Attitude Survey (ERAS; 
McKenna & Kear, 1990) indicated that students 
in these classes had relatively positive attitudes 
about in-school and out-of-school reading (see 
Table 1). On that instrument, all but two of the 
classes (Ms. Andrews at Collins and Ms. 
Sievers at Borders School) had mean raw 
scores that fell above the 50th percentile. 

An indication of students’ baseline 
levels of independent reading outside of 
school from the perspective of their parents 
can be found in a questionnaire completed 
by parents before the project began. The 
questionnaire is included as Appendix C; a 
key to scoring the questionnaire is shown in 
Figure 11; and, the means and standard 
deviations for selected variables are shown 



in Table 2. Of the variables listed, “range of 
reading” (see Table 2) may be considered a 
broad indicator of students’ levels of inde- 
pendent reading outside of school. The 
means reflect the average values across all 
students in each class whose parents rated 
them on 5-point likert scales associated with 
a variety of reading activities outside of 
school (item 9 on the questionnaire; see 
Appendix C). The mean values for each 
class range from 2.73 to 3.15 indicating 
moderate levels of reading outside of school, 
as perceived by parents, with little variation 
across the 9 classrooms. 

Teachers’ views toward teaching read- 
ing were determined in part through their 
responses on the TORP. Results of their 
responses to 5-point likert scale items are 
shown in Table 3, which lists means and 
standard deviations by individual teacher 
across the TORP’s three orientations to 
teaching reading: phonics, skills, and whole 
language. The range of scores across all 
teachers and orientation scales on the TORP 
completed before the project began was 2.3 
to 3.8 and the maximum difference among 
the scales for an individual teacher was 1.4. 
These values do not indicate strong differ- 
ences in orientations to teaching reading 
among the teachers nor an especially strong 
commitment to any one of the orientations 
by a particular teacher. These results are 
consistent with our observations of and discus- 
sions with the teachers. A notable exception 
is Ms. Andrews who expressed a strong 
commitment to whole language but whose 
responses on the TORP were weighted more 
heavily toward phonics and skills. 
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Variable Name 
(Range) 


Items 


Scoring 


1 . Free Time 
(0-2) 


1-2 


0 = reading not checked 

1 = reading checked 

2 = reading circled 


2. Reading/TV 
(x/y) 


2-3 


ratio of hours (nearest 1) 
reading (x) to hours viewing TV (y) 


3. Reading for enjoyment 
(0-4) 


4 


0 = rarely 
4 = all the time 


4. Ability estimate 
(0-4) 


6 


well above = 4 
well below = 0 


5. Children’s books 
(0-5) 


7 


none = 0 
fewer than 5 = 1 
more than 100 = 5 


6. Range of reading 
(x/22) 


9 


sum of all responses on 1-5 scale (x) 
divided by 22 (scores for 3 items reversed) 


7. Library card 
d-2) 


11 


no = 1 ; yes = 2 


8. Trips to library 
(0-6) 


12 


>5 = 6 


9. Books checked out 
(0-3) 


13 


none = 0 
1 to 3 = 1 
4 to 10 = 2 
> 10 = 3 


10. Change in reading 
(0-4) 


19* 


0 = much less 
4 = much more 


1 1 . Change in interest 


20* 


0 = much less 
4 = much more 



*Items included only on postexperimental questionnaire 

Figure 11. Key to variables on parent’s questionnaire. 
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Technology and Students ’ Interactions 
With Peers and Teachers 

Analysis of our field notes and videotapes 
of students and teachers in the computer lab 
documented consistently that peer interaction 
was greater during times devoted to the multi- 
media book review activity than during times 
devoted to other academic activities. This 
finding is consistent with Dickinson’s (1986) 
ethnographic study of collaborative writing 
involving computers in primary-grade class- 
rooms. However, because formative experi- 
ments focus on how an instructional interven- 
tion contributes to the accomplishment of a 
pedagogical goal, we were interested in how 
the increased peer reaction might relate to 
students’ reading. Further analysis revealed 
that increased peer interactions mediated the 
effect of multimedia book reviews on students’ 
independent reading. This mediation is illus- 
trated by the following event transcribed from 
tape recorded observations at Collins School: 

February 16 

I worked with Aaron, Dee, and Tyrone 3 to 
show them how to add audio to their basic 
template [on the computer], I asked them 
who had a book we could use as an example. 
Aaron said he had Where the Red Fern Grows 
and got it from his desk. He told some of the 
story into the microphone and I showed how 
his comments could be recorded while the 
others observed. Dee asked Aaron if she 
could look at the book after we had finished. 



3 A11 students’ names are pseudonyms throughout this 
report. The three students in this incident were focal 
students. 



February 23 

I saw the book Where the Red Fern Grows 
on Dee’s desk and asked her if she was read- 
ing it. She said that she was and that she 
liked it. When I asked her why she picked it 
to read, she said that she didn’t know why 
and didn’t seem to remember the incident 
with Aaron. 

Similarly, Ms. Pearson at Hartwig School 
wrote in her log in January a comment that 
coincidentally refers to the same book: “Beth 
Rollins read ‘Where the Read Fern Grows.’ 
Most of the people in that section of the com- 
puter lab have now read that book.” 

As represented by these incidents, we found 
that contending with the technological skills 
necessary to create multimedia book reviews 
led to increased peer interactions, which often 
led in turn to incidental sharing of information 
about books, and, for some students, to more 
reading. This finding is important because it 
was not anticipated to occur in the early stages 
of implementing the intervention. Originally, 
we were interested in moving quickly through 
the necessary technological training so that 
students could enter book reviews into a data- 
base. Our prediction was that the database 
would provide a mechanism for students to 
exchange information about their reading and 
thus promote more interest in reading a wide 
variety of books. Despite the fact that we were 
not able to implement fully the database com- 
ponent as planned, we unexpectedly found 
evidence that learning technological skills led 
to incidental sharing of information about 
books. 
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In addition, the interactions we observed 
during the times students were working on 
multimedia book reviews in the computer lab 
were different from their interactions with 
peers and with teachers in the classroom. Two 
related differences are illustrated in an event at 
Borders School. We were working with the 
teachers there to select some video clips to use 
in a presentation they were developing for a 
state language arts conference. Ms. Ellers 
selected a clip of two girls, one who performed 
well academically and one who performed 
poorly, working on their multimedia reviews in 
the computer lab on adjoining computers. 
Because the girls were clearly interacting about 
some aspect of entering their book reviews, 
this clip was chosen to illustrate student collab- 
oration and cooperation. The audio was of poor 
quality and our first assumption was that the 
high achieving girl was assisting the low 
achieving girl. However, after watching the 
clip several times and listening carefully to the 
audio, Ms. Ellers, with enthusiastic surprise, 
pointed out that the low achieving girl was 
actually offering the assistance. 

As illustrated by the previous event, stu- 
dents’ interactions related to entering multi- 
media book reviews were more cooperative and 
collaborative in the computer lab. Dealing with 
the challenges of learning the technological 
skills necessary to enter multimedia book 
reviews seemed to generate a heightened sense 
of camaraderie and helpfulness among stu- 
dents. In the computer lab, they seemed genu- 
inely interested in the achievements of their 
classmates and in the products they were 
developing. Discovering special effects on 
the computer screen or creative applications 



of HyperCard tools were often cause for 
special attention and sharing. For example, at 
Hartwig School, Betty, a student who had a 
Macintosh computer she could use at home, 
created a special presentation on the computer 
as a Valentine to Ms. Pearson, her teacher. 
The whole class gathered around a computer to 
enjoy her presentation, several students asking, 
“How did you do that?” Helpful interactions 
tended to occur spontaneously; but at Hartwig 
School, we (the teachers and the university 
researchers) tried to enhance this positive effect 
by asking students who questioned us about 
some aspect of HyperCard to consult with 
another student who could help them. Across 
all of the classrooms we observed, negative 
interactions occurred less frequently in the 
computer lab and they tended to be conflicts 
related to using the technology such as a dis- 
pute about whose turn it was to type at the 
keyboard. 

Another difference illustrated by the inci- 
dent on the videotape was that, in the computer 
lab, helpful interactions originated from the 
lower achieving students as often as from the 
higher achieving students. Students who had 
academic difficulties in the classroom some- 
times became experts in the computer lab 
where they might be called upon to assist their 
higher achieving peers. For example, our field 
notes taken while observing Shawn, a focal 
student classified as low reading ability and 
interest, state, 

Shawn has been ready to assist others in 

reading what was on their screens in order 

to figure out what should be done next; 

and he was not at all inhibited about 
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helping someone new to the class with 
what needed to be done. 

As this example indicates, the assistance of- 
fered by students having low reading ability 
frequently involved reading and interpreting 
texts. 

Interactions between teachers and students 
were also affected by participation in the multi- 
media book review activities. For example, it 
was clear that teachers did not define them- 
selves as experts when presenting information 
related to the multimedia book review activi- 
ties. Neither did students hesitate to offer their 
expertise to teachers. In fact, all of the teachers 
involved in this project seemed to enjoy the 
idea that many students exceeded their own 
technological understandings and that students 
were often able to teach their teachers. The 
following statement by Ms. Ellers’ from an 
October meeting of the teachers and research 
team illustrates this viewpoint: 

. . . and they love to show me because 
even with the demonstration this week of 
the new fields, you know ( gave rapid 
directions on how to do fields), well then 
I said you need to do that each time. Well 
then today, and this was the first time that 
I even knew it, they said ‘oh you don’t 
have to do that for a new field, you just go 
up to new field and automatically go over 
there on the tool palette.’ Like wow 
(laughing). I didn’t know that. But hey, 
they really get a kick out of that. 

Likewise, Ms. Palmer at Hartwig School in a 
semistructured interview about her experiences 
with and attitudes about computers in the 



classroom stated, “It doesn’t bother me at all to 
say to one of my children to tell me how you 
did that (on the computer).” 

We did not find any evidence that the num- 
ber of computers had any noticeable effect on 
the level of peer interactions. At Collins school 
where there was a single computer in each 
classroom, peer interactions related to the 
multimedia book reviews occurred over an 
extended period of time as small groups of 
students took turns using the computer during 
free time. At Collins and Hartwig Schools, two 
or three students often worked together at the 
computer. However, at Borders School, where 
students typically worked at their own com- 
puter, there was still considerable interaction 
among the students working in the computer 
lab. Students would be curious about each 
other’s work or seek assistance, which led to 
considerable interaction. 

The connection between increased peer 
interaction and incidental sharing of informa- 
tion about books illustrates how findings that 
emerge during a formative experiment can 
guide adjustments in implementing the inter- 
vention. For example, at Hartwig School we 
tried to enhance the effect of increased interac- 
tions as they related to books by pairing stu- 
dents to debug HyperCard stacks created by 
students in another class. Students completed 
this activity periodically using a guide sheet 
that we provided for them. We reasoned that in 
the process of debugging the programs, paired 
students might also interact with each other 
about the books being reviewed by their peers 
in other classes. Interestingly, we did not find 
that to be the case. In fact, the guide sheet 
tended to encourage students to take on the role 
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of highly critical editors who looked for sur- 
face level mistakes in spelling and punctuation. 
Likewise, when we purposefully used students’ 
work from Collins School as examples to 
students in the Hartwig School, the Hartwig 
students often focused on surface level mistakes 
in the examples. We did find some evidence 
that this critical stance did enhance students’ 
concern for the accuracy of their own work. 
However, because students’ reviewing of each 
others’ work seemed to distract students from 
activities directly related to the experiment’s 
pedagogical goal, we did not introduce this 
activity at Borders School during Year 2. 

Effects Related to Reading Ability 

From our data, especially related to focal 
students, we noted that reading ability often 
figured prominently in determining and under- 
standing the effects of the multimedia book 
review activity, although these effects were 
complex and often varied across classrooms. 
For example, Ms. Broward’s class at Collins 
School had a disproportionately high number of 
readers reading well below grade level when 
compared to other classes in her school. She 
often expressed concern about students’ lack of 
achievement and communicated her displeasure 
to students when she felt they were not meeting 
their potential. In her log, Ms. Broward noted 
that many of the poor readers in her class 
did not seem interested in entering book 
reviews. In a focused group interview with 
her students where we were seeking students’ 
reactions to entering their books on the com- 
puter, James, one of the focal students classi- 
fied as a poor reader, stated, “All I can read 



is easy books anyway, so why put them on 
the computer.” 

We hypothesized that some of the poor 
readers, in Ms. Broward’s class may feel the 
same way; that is, perhaps they were embar- 
rassed to enter books below grade level, thus 
publicizing their reading problems. We met 
with Ms. Broward to discuss this possibility. 
She agreed that it might explain the lack of 
interest among the poor readers, and we dis- 
cussed possible solutions. Ms. Broward sug- 
gested that since we had discussed eventually 
making the database containing her students’ 
reviews available for students in lower grades 
to find books they might like to read, it would 
be necessary to have easier books in the data- 
base. 

This observation evolved into a relatively 
minor but effective formative adjustment in the 
way the intervention was implemented in Ms. 
Broward’s class. Soon after our meeting, Ms. 
Broward announced to the class that she was 
concerned about the lack of multimedia book 
reviews of easier books. She explained that 
eventually some second- and third-grade stu- 
dents would be interested in searching the 
multimedia book reviews for books they might 
like to read. She expressed hope that some 
students would enter easier books for these 
students. Following this announcement, we 
noted an increase in the number of book re- 
views entered by poor readers. In fact, within 
several weeks, James had entered more books 
than any other student in the class. Because he 
tended to be a leader, at least among his poor- 
reading peers, his sanctioning of the activity 
seemed to encourage other poor readers to 
enter easy books too. 
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On the other hand, in Hartwig and Borders 
Schools where we taught students basic Hyper- 
Card skills, the low achieving students seemed 
to gain confidence and self-esteem immediately 
from working on the computers. For example, 
Robert, a focal student in special education at 
Hartwig school, one day presented a member 
of the university research team with an illus- 
trated poster printout he had made on the 
computer that stated, “I love computers.” 
Follow-up discussions with his classroom 
teacher and special education teacher revealed 
that Robert was enjoying his new status as a 
computer expert not only in the lab but also 
showing others how to use the computer for 
word processing in his classroom. Ms. Pear- 
son, his teacher, wrote in her log, “When we 
got a word processor to use, Roger was the one 
who showed the rest of the class how to use it. 
He was proud of himself, and so was I.” 

There is some evidence that Roger’s atti- 
tudes toward reading improved as well. His 
overall score on the Elementary Reading Atti- 
tude Survey (ERAS) increased from the 62nd 
percentile in September to the 78th percentile 
in the following May, although there was little 
evidence that he was reading more or more 
divergently as measured by the Parent’s Ques- 
tionnaire and the Choosing Things to Read 
instrument. Interestingly, his higher overall 
score on the ERAS was the result of a substan- 
tial increase on his score for the school reading 
subtest that compensated for a slight decrease 
in his home reading score. 

Related to our finding that dealing with 
technological issues increased peer interac- 
tions across academic ability, we found that 
dealing with the more complex technological 



issues related to creating multimedia book 
reviews tended to obscure visible differences 
in reading ability, which were more obvious 
when students were engaged in other aca- 
demic activities. For example, we attempted 
to validate independently through our obser- 
vations teachers’ selections of focal students 
who varied on reading ability and interest. It 
was not possible to do so by only observing 
students working with the computers, as 
indicated by these notes from our October 
field notes while observing in the computer 
lab: 

Q (to myself) What about 4 selected 
students? I can’t tell who they might be in 
here ... Q (to myself) Does anyone 
appear to have a problem with reading? 
Doesn’t look like it in here. 

On the other hand, it was possible to do so 
by observing students in the classroom, as 
indicated again from our October field notes: 
“Aide [in classroom] assisting students who are 
having trouble with their work. Teacher moni- 
toring/checking students comprehension— esp. 
poor students.” The teachers’ comments and 
our own observations repeatedly converged to 
indicate that distinctions in reading ability were 
less noticeable in the computer lab than in the 
classroom. 

Shane, one of the focal students classified 
as low ability and low interest in reading, was 
observed to have many difficulties reading in 
his classroom, but had no problems reading in 
the computer lab. We observed him avoiding 
reading and answering questions in the class- 
room but not in the computer lab when work- 
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ing on the multimedia book review activities. 
In an interview, Ms. Ellers stated: 

Shane . . . [was] really loving the initial 
stuff where I would show the whole group 
what we would be doing on Monday. With 
their little group, I would show them on 
the overhead and they would see what we 
were doing that day. And he would re- 
member it all. He was very successful 
when we were just creating their stacks 
and doing the fields and buttons and copy- 
ing from Art Bits {clip art available for 
students to use in their reviews) and what- 
not. 

Lower ability students who were not especially 
adept at using the technology could distinguish 
themselves in other ways, too. For example, 
Ms. Burton states in her log, “Even my slow 
ones are doing a good job. Lee and David 
(special ed.) have some good ideas for creating 
stacks. They need more help, but other stu- 
dents are there to help out.” 

The multimedia book review activities also 
tended to stimulate creativity among many of 
the high ability readers; in our estimation, 
more so than would be likely with conventional 
book reports. And, it seemed to reinforce their 
typically strong interest in reading. For exam- 
ple, Elizabeth (E), a focal student identified as 
having high ability and interest in reading 
responded to a researcher’s (R) interview 
questions as follows: 

R : Have you told anyone about this HyperCard 

project? 

E : I’ve told basically friends and family. 

R: What have you told them? 



E : I told them all the things that we had been 

doing that day in the computer lab, and how I 
couldn’t wait until the next time we go in 
there. It’s real fun. 

( discussion about the previous HyperCard 
lesson on audio buttons) 

E: That was fun. I did mine. I did all my voice 

impressions. I did my Zsa Zsa Gabor impres- 
sions, I did a real sophisticated lady impres- 
sion, and Mrs. Morris was saying, Go on, 
keep going. 

Likewise, in a February interview Ms. Ellers 
commented on the fact that Elizabeth has 
always been a good reader, while citing an 
example of how working on the multimedia 
book reviews provided an opportunity for 
recommending a book to her teacher. 

Elizabeth is a very top-notch student. She 
reads a lot anyway. She even brought me 
a book before Christmas and said you 
need to read this. I said I would try to 
read this over the holidays. She was in my 
third-grade classroom and we’ve been real 
close, and so she is comfortable with me. 

Of course she is doing well and I can see 
an increase in reading. She has always 
been a top reader. 

Transcriptions of a tape-recorded April 
meeting of the university research team and the 
project teachers aimed at discussing how to 
increase students’ sharing information about 
books revealed that good readers were already 
doing this: 
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R: Could we have the students to exchange book 

reviews . . . ? 

Ms. E : They’ve already done that. One’ll hear 
what another says, or they will tell me 
about their books— but these are the top 
students. 

Similarly, Ms. Palmer at Hartwig School wrote 
in her journal, “I have found that students that 
I find that read consistently were the ones that 
had several books to put into the files after 
Christmas.” 

Increased Student Engagement 

Our data collected in various ways and 
across various contexts contain repeated exam- 
ples indicating that students were more engaged 
in learning and using the technology related to 
creating multimedia book reviews than in other 
academic activities in the classroom. A typical 
example comes from a comparison of our field 
notes while observing Shane in the classroom 
and in the computer lab: 

{notes from classroom observation) 

Shane [5th grade focal student— low abil- 
ity/low interest] does not appear to be 
working at all on his math problems. . . . 
Shane is turned around in his seat to talk 
to his neighbors. . . . Shane is the first 
one done with his math. . . . Shane is 
sitting and tapping some pencils. Shane 
is looking at what a neighbor is drawing. 

. . . Shane is playing with two blue high- 
lighters. . . . Shane gets up to check the 
lunch menu. . . . Teacher asks Shane 
about whether he has shown his work, and 



where his work page is. Shane answers 
that he has thrown away his work. Teacher 
says to add the work on to his paper. . . . 

I haven’t been looking, but Shane appears 
to be done again (no one else is). 

{notes from computer lab observation , two 
days later) 

Shane passes out the template guide at the 
beginning of class. ... All students work- 
ing now; no one off task. A new students 
gasps. Shane (in a neighboring seat) says, 
“What’s wrong?” Antoinette says, “I 
don’t know.” Shane immediately scoots 
over to help her. . . . 

{later that class period) 

Antionette : “You left out your ‘E’.” 

Shane : “Where?” 

Another illustration comes from our video- 
tapes of students working in the lab. Ms. Ellers 
and a member of the research team discovered 
that Jason, a student whom teachers considered 
to be hyperactive because of his extreme dis- 
tractibility in the classroom, was sitting almost 
motionless staring at the computer screen for 
more than a minute working on a multimedia 
book review activity. Most of the obvious 
examples of increased attention were related to 
students who were poor readers. For example, 
in discussing a poor reader at an October 
meeting between the teachers and the univer- 
sity research team, Ms. Ellers stated, 

Ms. E : I know this group of fifth graders real 
well. Some of them have just really ex- 
celled, ’cause I bragged on one of them in 
[Mrs. Morrisfs room the other day be- 
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cause this child is probably reading at least 
two years behind grade level; but when he 
comes in here, he’s totally tuned in to what 
I’m doing, and he probably cannot read. 
It’s probably accurate to say he cannot 
read, you know, the pull down menus and 
what it says. But he is focusing exactly on 
what I’m doing, he’s seeing where I’m 
dragging down to, and I mean, he’s going 
to town on it. . . . He copied five pictures 
when everybody else, you know the most 
was two. 

Beyond increased attention and involve- 
ment in tasks related to the multimedia book 
review activities, increased student engagement 
was a label that captured other categories of 
our observations as well. For example, many 
students seemed to acquire a different persona 
when involved with project activities, often 
becoming less inhibited, more verbal, and 
more cooperative. In an audiotaped focused 
group discussion with the teachers, Ms. Morris 
observed: 

And a lot of the ones that answer ques- 
tions in the media center, like when we’re 
doing the whole group; a lot of those that 
I’ve seen, you know they raise their hands 
and they wanna answer are those that you 
wouldn’t, they’re not real verbal in class 
as far as answering something that we’ve 
discussed. You know, they’re the ones 
that remember that you {several voices, 
unintelligible) edit and go down to this, 
and what to punch to do this. 

Students who were shy and reserved in the 
classroom seemed more willing to take risks 
and ask questions when involved in creating 



multimedia book reviews as indicated from the 
following reflection from our field notes. 

I find it interesting that, having watched, 
that they WILL ask questions. Because 
there are some students, and I’m sure you 
know it, that if they don’t understand, 
they won’t even ask questions. But in 
here, it seems that nobody’s afraid to ask 
something, you know if they don’t see 
something right. I have no idea of their 
abilities, because I’m not there every 
single day to see who’s the best and who’s 
the worst, or not even to use those terms. 

The technological challenges of using 
HyperCard seemed to enhance students’ en- 
gagement as opposed to frustrating them. As 
Ms. Sievers stated in a focused group meeting 
with teachers, “. . . I haven’t seen anybody get 
really frustrated. They do in the classroom on 
other things, but not on this. ” In fact, students’ 
frustration surfaced mainly if something pre- 
vented them from working on the computers. 
Ms. Sievers observed in the same meeting. 



[the students] just threw a fit if they didn’t 
get to come to HyperCard. I mean, . . . 
we had thought it might be a little late, 
was it yesterday? {looking at Ms. Ellers) 
and I mean we fell to pieces on the floor 
in the classroom. They really got upset 
because they didn’t think they were gonna 
come, and when I tried to explain, I mean 
they were just so [upset]. 



When we talked to students about their 
participation in the multimedia book review 
activities, their responses were almost always 
positive reflecting their motivation and engage- 
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ment. However, their positive responses were 
almost always related to using the computer, 
particularly HyperCard, as opposed to specific 
comments about book reviews. Typical com- 
ments were, “It gives you a chance to do more 
activities and funner stuff,” . it’s like; you 
can draw . . . you can pick pictures to draw 
and stuff. You can do your own voice and it 
(HyperCard) has all sorts of stuff on it,” 
“’cause you get to draw and don’t do no work 
. . . (Researcher asks: HyperCard’s not 
work?). . . . Nope.” In fact, as students mas- 
tered HyperCard skills and began to focus 
more on entering book reviews, some of the 
motivation and enthusiasm waned. In a Febru- 
ary meeting with teachers aimed at discussing 
formative adjustments in the way we were 
implementing project activities, Ms. Ellers 
stated, 



. . . they are coming in there quite regu- 
larly (students who have read books to put 
on the computer), so they’re really pro- 
gressing whereas some kind of get left 
behind. We haven’t decided on how we’re 
going to handle getting those others to get 
to that state— you know, the less motivat- 
ed— that’s something we need to sit down 
and discuss . . . 



At times, the lure of nonacademic games and 
drawing programs also overshadowed students’ 
interest in entering book reviews. For example, 
Ms. Palmer at Hartwig School wrote in her 
journal that one of her students suggested “a 
nonstructured day in the lab, just to have fun 
... as a break.” 



Our discussions of formative adjustments 
to implementing the multimedia book review 
activities often centered on how to channel 
students’ increased engagement related to 
HyperCard and the computer toward accom- 
plishing our pedagogical goal. At times there 
were tradeoffs, sometimes due to technological 
considerations. For example, the decision to 
use a standard template for the book reviews to 
be searched in the database, decreased the need 
for students to exercise their HyperCard skills. 
We compromised by adding an open-ended 
section to the template where students could 
use their HyperCard skills to create additional 
individualized information about each book 
they reviewed. At Hartwig School, the teachers 
decided to insist that students enter new infor- 
mation about the books they were reading 
during the first 20 min of the weekly hour-long 
session in the computer lab; during the remain- 
ing time, students were free to incorporate 
multimedia presentations into existing book 
reviews if they chose to do so. However, it is 
important to note that we did not find any 
evidence that students disliked entering book 
reviews or that the intriguing aspects of using 
multimedia on the computer distracted students 
from reading books. 

Effects Related to Particular 
Schools and Teachers 

As expected, the specific effects of the 
multimedia book review activities varied 
among the schools and classrooms participating 
in the project. Working in seven classrooms in 
three schools with seven classroom teachers 
and two key adjunct teachers allowed us to 
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discover general effects of the intervention that 
cut across the different instructional contexts, 
but also allowed us to speculate about varia- 
tions that intensified or mitigated those effects. 
In this section, we discuss these variations. Our 
findings in this area are limited by the fact that 
the intervention was modified when imple- 
mented during Year 2 in Borders School based 
on what occurred during Year 1 in Collins and 
Hartwig Schools. In a more general sense, any 
formative experiment is limited by conditions 
that are not amenable to adjustment in particu- 
lar situations, but noting the effects of these 
conditions may enhance understanding of their 
importance in achieving the pedagogical goal. 

Examples of conditions that were not easily 
modified in the present formative experiment 
include the impractical ity of creating multiple 
versions of the software, the availability of 
space and equipment in a particular school, 
scheduling students, and so forth. Beyond such 
practical, technological, and logistical con- 
straints, there are the expected differences in 
teaching styles, educational values, instruc- 
tional approaches, and so forth that vary among 
teachers and schools. Even if it were possible 
to modify these naturally occurring differences 
in educational orientation, any attempts to do 
so are fraught with ethical problems. Thus, the 
differences we discuss in this section are aimed 
at understanding more about the effects of the 
present intervention, but our specific examples 
may also be generically useful in understanding 
issues researchers may face when conducting 
formative experiments. 

We devoted considerable attention to 
searching our data and speculating about causes 
for the clear differences we observed between 



Collins and Hartwig Schools during Year 1. 
Teachers at both schools were enthusiastic 
about the project in the early stages, as were 
members of the university research team. For 
example, as the project began, teachers in both 
schools wrote enthusiastic comments in their 
logs indicating that they had high expectations 
for the project and were looking forward to 
their participation. During an interview early 
in the school year before the computer activi- 
ties were introduced, teachers indicated that 
they thought the multimedia book review 
activities would “inspire students to read,” 
“allow students to use the computer in an 
interesting way,” and “give me [the teacher] an 
opportunity to learn more about the computer. ” 

As the year progressed, however, the clear 
enthusiasm and morale evidenced early in the 
year deteriorated at Collins School but not at 
Hartwig School. By spring, Ms. Andrews at 
Collins School, who initially had greater com- 
petence and parental assistance for using com- 
puters and higher achieving students in her 
classroom than did her colleague Ms. Broward, 
had fallen far behind her, almost abandoning 
entirely the project activities. At Hartwig 
School (and later at Borders School), project 
activities extended into other areas of the 
curriculum, but this was not the case at Collins 
School. 

Drawing on our observational and inter- 
view data, the following factors seem to be 
involved in explaining the difference between 
these two schools: 

1. The professional climate of the two 
schools was distinctly different. For example, 
although principals and central office personnel 
were highly supportive of the project in both 
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schools, the administrative style in Collins 
School was more top down, and administrators’ 
interest in the project seemed related more to 
its value for public relations in the community 
and its implications for test scores than for its 
potential to enhance curricular goals or to 
further students’ and teachers’ development. 
Teachers seemed constrained, pressured, and 
sometimes even intimidated by this administra- 
tive stance. We conducted interviews with six 
teachers at Collins School early in the school 
year. When asked about possible negative 
outcomes of the project, virtually all of them 
expressed concern that they might not have 
time to work the project activities into their set 
schedules. For example, Ms. Andrews justified 
her preference for using computers in a lab 
setting by stating that “the lab guarantees 
everybody the right to 45 minutes.” (Although 
teachers at the other schools also frequently 
discussed the pressures they were under to 
cover the established curriculum, they were 
willing to seek out ways to work the computer 
activities into their schedules and commented 
on how worthwhile it was to do so, and they 
seemed to expect cooperation from their ad- 
ministrators.) Additionally, toward the end of 
the school year, to our great discomfort, it 
became evident that administrators at Collins 
School were not honoring teachers’ requests 
for future teaching assignments because of the 
new assignments’ potential effect on the proj- 
ect. In contrast, administrators at Hartwig 
School seemed focused more on student and 
teacher development, respecting teachers’ 
judgement in altering their schedules when 
necessary and encouraging them to work as an 
team, seeking administrative support when 



necessary. For example, a visit from the Dis- 
trict Language Arts Coordinator at Hartwig 
School was punctuated by many comments to 
members of the university research team 
about the project’s potential to promote the 
District’s goal of moving toward a more an 
integrated curriculum rich in authentic 
reading and the goal of enhancing teachers’ 
technological expertise. Although adminis- 
trators associated with Collins School occa- 
sionally referred to similar goals, they more 
often mentioned how the project would 
promote the school’s reputation such as 
achieving a state-wide designation as a 
school of excellence. 

2. The Collins teachers perceived that they 
were not getting as much attention from the 
university research team as were the Hartwig 
teachers. Furthermore, they perceived that 
when compared to the Hartwig teachers they 
were at a disadvantage because they only had 
one computer in their classroom instead of a 
computer lab. The event that clearly initiated 
these perceptions was a joint presentation 
involving teachers from both schools at a 
national conference. During the presentation, 
Collins teachers saw that they had not accom- 
plished as much as the Hartwig teachers, and 
from that point seemed dissatisfied with their 
own situation as evidenced by frequent refer- 
ences to Hartwig’s superior resources during 
our subsequent interactions with them. To a 
certain degree their perceptions were accurate. 
The research team did spend more time at 
Hartwig because more time was necessary to 
teach students how to use HyperCard, whereas 
we delivered more finished products to Collins. 
The substantially greater distance to Collins 
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School also limited the number of visits by the 
university research team. As the year pro- 
gressed, we found it increasingly necessary to 
give teachers at Collins encouragement that we 
were not disappointed with their progress, yet 
this encouragement became less convincing to 
the teachers, and importantly to ourselves too 
as the year progressed. Nonetheless, we be- 
lieve the differences in morale are due more to 
the teachers’ perceptions than differences in 
resources or support. Neither are we convinced 
that the multimedia book review activity faces 
insurmountable problems in a classroom with 
only a single computer. 

3. Teachers at Collins School seemed to be 
more conscious of whether the research was 
being conducted properly and whether the 
activities were successfully meeting our goals. 
We explained the concept of a formative exper- 
iment to teachers at both schools early in the 
project emphasizing that we did not expect the 
project activities to succeed uniformly in every 
context or without formative adjustments based 
on our data collection. Regardless of the 
school, teachers on the whole did not often 
seem comfortable with pointing out difficulties 
with project activities or with suggesting ideas 
for improvement with members of the university 
research team, to whom they often deferred. 
For example, during January, Ms. Pearson at 
Hartwig School wrote in her journal: 

I had my interview with Dr. R today. I 
was very relieved to find that they [the 
university research team] were pleased 
with our progress. A lot of apprehension 
was relieved. 



However, the Collins teachers seemed to be 
more concerned about how the data were being 
collected, whether we thought the project was 
being successful in our eyes, and whether we 
approved of their involvement. Their concern 
and doubt may have reflected their decreasing 
morale and likewise exacerbated it. Conse- 
quently, the Collins teachers seemed to rely 
heavily on our direction and support. Ideas for 
extending the project were discussed with 
enthusiasm but were not often implemented 
without direct support and followup from the 
university research team. 

4. Although teachers in both schools 
expressed that they felt pressured to cover 
curriculum, Ms. Andrews and Ms. Broward at 
Collins School were under greater stress from 
other sources than were the teachers at Hartwig 
School. In addition to greater administrative 
pressures to conform and achieve, both of their 
families experienced serious medical problems 
during the school year. In addition, at Collins 
we did not observe a strong sense of teamwork 
and mutual support among the teachers at the 
school. Ms. Andrews and Ms. Broward origi- 
nally volunteered to become involved in the 
project because they had developed an affinity 
based on their interest in whole language, 
openness to new ideas, and desire to learn 
more about technology. Throughout the project 
they clearly collaborated well on project activi- 
ties under the direction of the university re- 
search team, but they did not fully integrate 
their efforts, at least to the extent we observed 
at Hartwig School. There was little sharing of 
expertise and resources that proceeded from 
joint problem solving. In our interviews and 
informal interactions with other teachers at the 
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school, we often heard of criticisms, jealousies, 
and disagreements between and among teach- 
ers, some of which extended to incidents or 
situations outside of school in the community. 
In addition, there was noticeable turmoil in 
Collins School precipitated by the fact that the 
principal was running for election as school 
superintendent, there was uncertainty about 
who would be her successor as principal when 
she was elected in November, and which 
teachers would be assigned to a new school that 
was to be opened the following year. Collins 
School was also preparing for an extensive 
remodeling that disrupted activities, especially 
during the last month of the school year. 

5. The fact that the project activities were 
carried out with a computer in the classroom at 
Collins as compared to a computer lab in the 
other schools may have been a factor that 
figured in the differences we observed. Sched- 
ules had to be arranged to use the computer 
labs, thus there needed to be a fixed time in the 
teachers’ schedules for engaging in book re- 
view activities, and that time had to be coordi- 
nated with the other teachers involved in the 
project. Having the flexibility to work the 
activities into one’s schedule on an on-going 
basis, as was the case at Collins, may have led 
to fewer collaborations between the teachers 
and the temptation to forego the activities when 
they felt pressures to complete mandated in- 
struction. Additionally, students at Collins 
seemed less enthused during group presenta- 
tions j perhaps because these presentations 
occurred in their classroom as opposed to 
making a special trip outside the classroom to 
the computer lab. For example, in our field 
notes recorded during a presentation about how 



to use the computer, we observed the follow- 
ing: “. . . there were several students who 
were not paying much attention ... it was 
snack time and some of them were exchanging 
potato chips with each other and others were 
preoccupied with an art project.” In contrast, 
our notes observing in the computer labs sug- 
gested that students were highly attentive to 
presentations unless distracted by their desire 
to work on the computer. 

Another difference between Collins School 
and the other two schools relates to variations 
among teachers and their roles in implementing 
the intervention. The effects of the multimedia 
book review activity were clearly influenced by 
the characteristics of individual teachers work- 
ing together with their colleagues and the 
research team on the project, although the 
effects of teacher variations and school varia- 
tions clearly interact. All of the teachers who 
participated in the project were enthusiastic 
about their participation at the outset, and none 
had more than minimal acquaintance with 
using computers personally and in the class- 
room prior to this project as evidenced by their 
responses to a structured interview (see Appen- 
dix A). However as the project proceeded, we 
found that teachers tended to fulfill several 
identifiable roles. To a certain degree, teachers 
shifted roles throughout the project, although 
they tended to gravitate toward one role. Each 
of the following roles seem salient in under- 
standing the effects of the multimedia book 
review activity and how it might be effectively 
implemented 4 : 



4 Our categories might be compared to those suggested by 
Hadley and Sheingold (1993). 
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The technology expert. A teacher at each of 
the three schools acquired this role early in the 
project by virtue of her greater interest and 
quicker success in mastering the hardware and 
software, accompanied by a greater commit- 
ment to work on the computer beyond the 
minimal requirements for the project. We 
conducted training sessions for all of the teach- 
ers prior to the beginning of the school year 
and the role of technology expert emerged at 
that time. The other teachers not only acknowl- 
edged this role (e.g., comments such as “Ms. 
Ellers will help us do that.”), but they seemed 
to need the technology expert to sustain their 
own efforts (e.g. , “I know I couldn’t have done 
that without Ms. Burton’s help.”), sometimes 
in a way that seemed to prevent them from 
extending their own technological expertise 
(e.g., “I couldn’t do this next year myself, but 
maybe I could with Ms. Burton’s and Ms. 
Pace’s help.”). The negative effects implied by 
this latter comment indicating dependency can 
be seen by Ms. Andrews progressive abdica- 
tion of the role of technology expert at Collins 
School. As Ms. Andrews became less involved 
in the project activities, Ms. Broward was 
forced to deal more with the technology herself 
and by the end of the school year had increased 
her expertise beyond Ms. Andrews. On the 
other hand, it was the teacher in the role of 
technology expert at each school who was 
instrumental in extending the project activities 
into other areas of the curriculum or other 
school activities, and this served as a model 
and resource for the other teachers. For exam- 
ple, Ms. Burton, who became a technology 
expert at Hartwig School, used her new Hyper- 
Card skills to create a tutorial directing stu- 



dents how to create a display for the school 
science fair. Likewise, Ms. Pace, a Chapter 1 
teacher assisting the teachers at Hartwig, 
quickly became a technology expert helping 
students with make-up activities and debugging 
their work. 

The emerging or marginal technology 
expert. Another role was characterized by 
teachers whose early involvement in the project 
activities was enthusiastic but passive, defer- 
ring almost entirely to members of the research 
team and to the teacher(s) in the role of tech- 
nology expert. Gradually, however, teachers 
in this category seemed to become more com- 
fortable using the technology and more enthu- 
siastic about the intervention’s effects. Ms. 
Pearson’s involvement during the project at 
Hartwig School is illustrative. Transcriptions 
of our tape recorded field notes early in the 
school year state, 

[Ms. Pearson] sat at the table away from 
the computers in the lab today counting 
money and filling out book orders. She 
seemed to put us [members of the univer- 
sity research team] in charge with no 
intention of participating in learning how 
to use the HyperCard stacks or of working 
with students. 

As the year progressed, however, Ms. Pear- 
son’s involvement increased greatly as she 
became more comfortable with the technology 
and saw the enthusiastic response of her stu- 
dents. Later in the year, on her own, she 
decided to extend the multimedia book review 
activity into her social studies unit on the Civil 
War. After all of her students had read a novel 
set at the time of the Civil War, she divided 
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her students into groups to develop a multime- 
dia presentation on some aspect of the book, 
including a class videotape in which students in 
costume acted out various episodes in the book. 
As the year progressed, Ms. Pearson became 
more actively involved in the activities in the 
lab by offering help to students or letting them 
teach her skills. Ms. Pearson did not seem to 
define herself as a technology expert, but did 
move far in that direction over the year. This 
growth is apparent from our field notes at the 
end of the school year: 

I told [Ms. Pearson] that next year we 
would be moving on to another school and 
that Hartwig teachers would be one their 
own. Of all the teachers at the beginning 
of the year, she is the one I thought would 
be most concerned with that. I thought it 
was significant that now she didn’t seem 
that concerned. She kind of made a little 
face when I told her, but it wasn’t at all a 
negative reaction . . . more like “I’m not 
sure” but maybe I can. 

Another example, was Ms. Morris at Borders 
School who in a teasing manner chastised us 
for not observing the day that she took over for 
Ms. Ellers and taught her students a lesson on 
using HyperCard. 

The facilitator . Some teachers, at least for 
a time, seemed to assume the role of a facilita- 
tor; that is, a teacher who is not especially 
intrigued with the technology, which was in the 
domain of the technology experts, but who is 
interested in discovering and enhancing the 
nontechnolog ical effects of the program. For 
example, Ms. Pearson’s movement toward 
technology seemed to originate with her at- 



tempts to facilitate the positive effects she was 
seeing on her students in the classroom. Ms. 
Sievers at Borders School also could be classi- 
fied as a facilitator because she did not indicate 
more than passing interest in the computer, but 
she attempted to connect classroom reading and 
writing activities with students’ multimedia 
book review activities in the computer lab. 
Likewise, teachers who focused on identifying 
and solving practical and logistical problems 
were assuming the role of facilitator. 

The passive participant. Teachers in this 
role may be enthusiastic about the project and 
its potential benefits, but they rely primarily on 
others (their fellow teachers or members of the 
research team) for explicit direction and guid- 
ance. Their personal investment in the project 
is low in terms of independent effort to engage 
in creative problem solving to address logisti- 
cal, practical, technological, and pedagogical 
problems. They do not contemplate possibili- 
ties for extending or adapting the multimedia 
book review activities or coordinating them 
with other curricular areas; or, if they do, such 
extensions must not create too much of a dis- 
ruption to their current instructional routines. 
There is little evidence among teachers assum- 
ing this role of attempts to master the computer 
outside of school hours including formal train- 
ing sessions led by the members of the univer- 
sity research team after school. They have a 
relatively low tolerance for dealing with devel- 
opments that prevent the activity from meeting 
their expectations of success. This role tended 
to be assumed more often by the teachers in 
Collins School, especially as the project pro- 
gressed. However, we wish to emphasize that 
this is not a generalization that extends to other 



NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55 



o 

ERIC 



61 



A Formative Experiment 



51 



instructional activities in their classrooms; nor, 
does it do justice to their more laudatory teach- 
ing practices. Additionally, we believe that the 
professional climate of their school and district 
may have predisposed them to assume this role, 
as did their declining morale explained previ- 
ously in this section. 

The Effects of the Project on the 
Educational Environment 



Our interest in the effects of the project on 
the educational environment was motivated by 
Newman’s (1990) explication of formative 
experiments. He states, 



Whatever the pedagogical theory moti- 
vating the experiment, the outcome to be 
observed must include how the environ- 
ment becomes organized differently as it 
appropriates the technology and other 
resources. ... If the environment, rather 
than the technology, is the unit of analy- 
sis, changes in the instructional interac- 
tions, changes in teacher roles, and other 
ways that the educational environment is 
changed are observed, (p. 10) 



We found that teachers’ involvement with 
project activities was somewhat paradoxical. 
Often they tended to view multimedia book 
reviews and their potential benefits in relation 
to conventional academic activities and skills. 
At the same time, particularly at Hartwig and 
Borders Schools, teachers were willing to give 
up time devoted to teaching conventional 
activities and skills so that students had time to 
work on the book review activities. In the 
teachers’ views, the multimedia book review 



activities seemed embedded within a conven- 
tional schema for what students ought to be 
doing and learning in school, while simulta- 
neously they engaged enthusiastically in an 
activity that to some extent subverted that 
schema. Teachers’ concerns that students 
acquire particular skills and be held account- 
able for their work co-existed with their enthu- 
siasm for activities that were clearly addressing 
less tangible goals (such as increasing students’ 
self-direction, confidence, creativity, and 
ability to master the technology). This dual 
focus can be seen in Ms. Burton’s log where 
she states. 



I felt they needed to see a stack in prog- 
ress to understand more about what is 
required. I want them to be creative! . . . 
Once they got into creating their own 
cards, the creative juices began to flow. 
[And one page later] I am having students 
record pages read each week and having 
parents sign. 



On one hand, teachers tended to see the 
project activities in conventional terms. For 
example, despite the research teams’ explana- 
tions that we preferred the term “book re- 
views” instead of “book reports” (because of 
the negative connotations of book reports) and 
despite our consistent use of this terminology, 
the teachers almost always referred to what 
students were doing as “multimedia book 
reports” or “book reports on the computer.” 
As described more fully in a subsequent sec- 
tion, teachers were especially enthusiastic 
about how the project activities were benefiting 
the technical aspects of students’ writing. Addi- 
tionally, they seemed more satisfied than did 
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members of the research team with conven- 
tional responses to books such as writing 
summaries as long as they were technically 
correct. For example, teachers were often 
reluctant to provide the research team with 
disks containing students’ work until they had 
been proofread and corrected, often by one of 
the teachers. In her log, Ms. Pearson stated, 
“They are not proficient writers. Their writing 
skills are very poor. They refuse to use dictio- 
naries to check their spelling. I am embarrassed 
to let some of them take their work to the 
computer lab. ” Some of the teachers sent home 
reports of students’ progress on the book 
review activity to parents and they were con- 
cerned that students not enter certain kinds of 
reading material such as magazine articles or 
joke books on the computer. We see these 
findings as consistent with previous research 
(e.g., Bruce & Rubin, 1993) indicating that 
teachers tend to conceptualize and use innova- 
tive activities involving technology in a way 
that conforms to conventional academic values 
and experiences regardless of the activities’ 
potential to transform standard practice. 

On the other hand, we saw some evidence 
that teachers’ enthusiasm for the benefits of 
the project led them to forego, displace, or 
extend more conventional activities in their 
classrooms. For example, although the 
teachers at Hartwig and Borders Schools 
often emphasized the many expectations 
placed upon them for covering content and 
becoming involved in special activities such 
as essay contests, they rarely opted out of 
the project activities even though we invited 
them to do so if necessary. In addition, 
teachers in both of these schools created 



their own plan for catching up students who 
had missed a lab activity. 

As the year progressed, at Hartwig School 
especially, we saw evidence that the teachers 
were appropriating computers and the book 
review activity into their teaching beyond the 
specific project activities. Contrary to the 
beginning of the year when computers in the 
classrooms were used almost exclusively for 
drill and practice or game programs for stu- 
dents who had completed their work, the 
classroom computers began to be used for 
other purposes. Ms. Burton, who initially 
acquired the role of technology expert at Hart- 
wig, used her new HyperCard skills to develop 
a tutorial directing students how to prepare 
their science fair displays . In our field notes we 
also observed that, 

[Ms. Pearson] was excited to show me 
that the students were working on a word 
processing station in her classroom. . . . 

She also told me that she had ordered a 
word processing program on the Apple 
[computer]. Apparently she did not have 
a word processing program in the class- 
room before. I think this may be an 
indication that she is starting to use the 
computer more in her classroom. 

Later in the year, one of us noted that, “I 
thought it was very significant that [Ms. Pear- 
son] told me in passing that she skips spelling 
to do word processing in her classroom.” 

From the standpoint of the project’s peda- 
gogical goal, the most notable example of how 
the project activities could extend and trans- 
form conventional instruction occurred in Ms. 
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Pearson’s class. In February, Ms. Pearson 
decided to break students into groups to create 
a class review on the computer of a book she 
had decided to have students read in conjunc- 
tion with a social studies unit on the Civil War. 
She wrote in her log, 



Today I gave all of my students a copy of 
Shades of Gray. They are going to do a 
book report on the computer. They were 
very excited and immediately began to 
read. I am going to let them group to- 
gether to do different parts of the book 
report. They are going to make the deci- 
sions on the way the book report should be 
done. Almost everyone uses any spare 
moment to read this book. 



Along with the computer-based activities de- 
signed by each group, the students with mini- 
mal teacher direction and support produced a 
video of episodes from the book. Students got 
involved in all aspects of the production includ- 
ing elaborate costumes and staging and accord- 
ing to Ms. Pearson, “They were willing to give 
up breaks to organize the video. ” The multime- 
dia book review project clearly led the teacher 
to conceive of this activity and for the students 
to carry it out with enthusiasm while becoming 
involved in a positive experience related to 
their reading. The activity also led students to 
explore independently aspects of books that 
they would have unlikely encountered prior to 
the intervention. For example, as Ms. Pearson 
states in her log, 



One group talked about the author and her 
personal life. They did research and 
discovered that the author was a Civil War 



history buff. She had actually visited and 
studied the Shanandoah Valley where the 
story takes place. 

Through this activity Ms. Pearson’s students 
also participated in a more in-depth response to 
a book than they would have in writing a 
conventional book review. For example, she 
wrote in her log that one of the groups in- 
volved in this class project dealt with “prob- 
lems of main character— How (problems) they 
were dealt with/book/how same problem 
should be dealt with by us if were faced with 
same challenges.” 

As the project progressed, teachers also 
noticed connections between classroom activi- 
ties and students’ work on the multimedia book 
reviews in the computer lab. At an October 
meeting, Ms. Ellers remarked. 

Well, these things that [John, a member of 
the university research team offering 
technological assistance] observed the 
other day though. . . . [W]hen after I 
show them what to do, I tend to go over 
here to my right and work with this 
group. And this group back here— there’s 
about three students. Well, they created 
their fields, but they started just typing a 
story in the fields. You know, they did 
their own little thing. So, you know, and 
he [John] said, “Well, where do you think 
they got that from?” And I said, “Well, 
they do a lot of creative writing in [Ms. 
Sievers’] class. So that has some [carry 
over from one situation to another]”. . . . 

Nonetheless, the general enthusiasm for 
and commitment to the project activities among 
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both teachers and students tended to balance, 
sometimes precipitously, on its relation to 
conventional instruction. Teachers often ex- 
pressed the benefits of the project in terms of 
conventional instructional goals while enjoying 
its less academically oriented effects, occasion- 
ally linking the two as when we pointedly 
asked the Borders teachers in a meeting: “Does 
the need to make the book review present- 
able — understandable — play any role in damp- 
ening students’ enthusiasm?” Both answered 
“no” loudly in unison. Students too were 
affected by this balance. If the multimedia book 
review activities began to take on the character- 
istics of conventional classroom activities, 
enthusiasm waned. For example, In a February 
interview Ms. Ellers observed, 



But several of the fifth graders when they 
come in here into the classroom [lab], 
they’re ready. They just need to plug in 
the information. That’s one biggie I would 
say. The fifth grade really likes doing it. 
I hear a lot of positive comments from 
them, from certain groups of them, be- 
cause there are several that don’t come in 
there as much anymore because they know 
that is the requirement now: if they don’t 
have their book review ready or written it, 
then they’re not ready to go to the lab. 
They’re going to miss their time whereas 
if someone else is ready that can go in 
their place since they now [each] have 
their own individual disk. And so some of 
them are in there two, three, or sometimes 
even four times a week. They are coming 
in there quite regularly, so they’re really 
progressing; whereas some kind of get left 
behind. We haven’t decided on how we’re 



going to handle getting those others to get 
to that state— you know, the less moti- 
vated— that’s something we need to sit 
down and discuss . . . 

Technology, particularly as manifested in 
learning and using HyperCard, seemed to play 
a pivotal role in tilting the balance away from 
conventional instruction. As documented previ- 
ously in this report, teachers were less inclined 
to see themselves as experts in the computer 
lab, often deferring to students’ expertise. In 
addition, students’ and teachers’ work on 
HyperCard involved highly engaging and 
interesting activities in a nonthreatening aca- 
demic environment that was separate from the 
classroom where participants carried out most 
of their daily routine. Students who had aca- 
demic problems in the classroom frequently 
excelled in using the technology, and students’ 
increased interactions with their peers and with 
their teachers were frequently supportive and 
positive. The cumulative effect of these charac- 
teristics seemed to override teachers’ concerns 
associated with conventional instruction. How- 
ever, entering information about books, as a 
more conventional activity, seemed to remind 
participants of more academic concerns, which 
in turn evoked more conventional reactions. 

As the emphasis on learning and using 
HyperCard decreased, concerns related to 
conventional academic goals and achievement 
increased, thus perhaps negating some of the 
potential influence of the multimedia book 
review activity. At Hartwig School this effect 
was mitigated by the fact that the intervention 
clearly extended into other classroom reading 
and writing activities and indeed into other 
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areas of the curriculum. At Collins School, 
where students were not systematically taught 
HyperCard and where students completed 
project activities in their classroom, the tech- 
nology remained firmly entrenched within the 
context of conventional classroom activities. 
This too, may account for the relative lack of 
involvement by teachers and students at Col- 
lins. We see Borders School as representing a 
middle ground where interest in technology 
was high, which sustained the activity much of 
the year, but where there was no clear evidence 
that the intervention was influencing instruc- 
tional activities beyond the specific book re- 
view activities. Consequently, some students 
seemed to lose interest in the book review 
activity later in the school year. 

Comparing our experiences within and 
among schools, we hypothesize that the initial 
contribution of the intervention toward accom- 
plishing the pedagogical goal is based on how 
involvement with technology, particularly 
learning HyperCard in a computer lab, estab- 
lishes a positive context that clearly transcends 
conventional academic activities, concerns, 
demands, responses, and so forth. Multimedia 
book reviews are associated positively with this 
environment and contribute to the pedagogical 
goal as mediated by factors such as increased 
interaction. Gradually, however, as emphasis 
shifts away from learning and using the tech- 
nology toward completing book reviews, a 
more conventional mindset emerges. Teachers 
begin to associate the project’s benefits with 
conventional academic skills such as writing 
skills and conventional practices such as 



requiring minimal amounts of reading. How- 
ever, these effects are mitigated when teachers 
more fully appropriate computers into their 
classroom teaching and extend the multimedia 
book review activities into other areas of the 
curriculum. These effects might also be miti- 
gated by moving quickly to the database phase 
of the intervention; although this is speculative 
since we were not able to implement fully this 
feature. In fact, the delay in implementing the 
database may have contributed greatly to the 
shift toward a more conventional perspective of 
the activity among some of the teachers. 

Quantitative data from the TORP com- 
pleted by the teachers at the beginning and 
again and the end of the school year are rele- 
vant to these issues. Pre- and postexperimental 
means and standard deviations on the TORP 
are shown by teacher in Table 3. Two statisti- 
cally significant changes in orientation to 
teaching were noted. Ms. Burton at Hartwig 
School became less oriented toward whole 
language as can be seen by comparing her 
means on that subscale at the beginning and the 
end of the school year. Ms. Ellers at Borders 
School became less oriented toward phonics 
as can be seen by comparing her means on 
that subscale at the beginning and the end of 
the school year. These data do not provide 
any clear evidence that the intervention 
changed teachers orientation toward teaching 
reading. Ms. Ellers’ move away from pho- 
nics may have been associated with the 
university course work she was taking as 
part of a degree program. Ms. Burton’s 
move away from whole language may have 
been due to the stress on skills associated 
with end-of-year testing. However, we have no 
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Table 3. Means (Standard Deviations) for Teachers’ Pre- and Postexperimental Scores on Subscales 
of the TORP 





Subscales 




Phonics (10 items) 


Skills (10 items) 


Whole Language (10 items) 


School/Teacher 


pre 


post 


pre 


post 


pre 


post 


Collins School 














Andrews 


3.8 a 


3.8 


3.8 


4.1 


2.4 


2.3 




(1.08) 


(1.02) 


(.84) 


(.90) 


(.71) 


(.75) 


Broward 


2.8 


2.8 


2.9 


2.3 


2.4 


1.8 




(.64) 


(.81) 


(.91) 


(.80) 


(.81) 


(.75) 


Hartwig School 














Burton 


2.9 


2.8 


2.5 


2.5 


3.0 


2.5* 




(1.14) 


(1.01) 


(.95) 


(.88) 


(.50) 


(.62) 


Palmer 


2.9 


3.0 


2.3 


2.4 


2.6 


2.4 




(1.30) 


(1.11) 


(.90) 


(.85) 


(.48) 


(.54) 


Pearson 


3.0 


2.9 


2.9 


2.6 


3.6 


3.2 




(.40) 


(.48) 


(.75) 


(.87) 


(.63) 


(.72) 


Borders School 














Ellers c 


3.2 


2.5* 


2.4 


2.3 


2.5 


2.6 




(.80) 


(.60) 


(.56) 


(.62) 


(.63) 


(.78) 


Sievers 


NA b 


3.1 


NA 


2.4 


NA 


3.9 






(.88) 




(.62) 




(1.05) 


Morris 


2.6 


2.5 


3.1 


3.1 


2.7 


2.8 




(.72) 


(.64) 


(.54) 


(.66) 


0.12) 


(1.30) 



a Means are based on a 5-point Likert scale where 1 = strong agreement; lower values indicate stronger 
orientation. 

b Pretest scores not available for Ms. Sievers because she replaced original teacher after baseline data were 
collected. 

Though Sievers took over Ellers’ class, Ellers was the teacher most directly involved in working with all 
students in the computer lab with each teacher. 

*p < .05. 
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supporting evidence from our qualitative data 
to clearly link the two statistically significant 
changes to the multimedia book review activities. 

Changes in the Amount and Diversity 
of Students ’ Reading 

The pedagogical goal of this formative 
experiment was to increase the amount and 
diversity of students’ independent reading. 
Before presenting our findings concerning the 
effects of the intervention on students’ indepen- 
dent reading, we discuss three factors that 
limited our ability to collect data aimed at 
determining whether the intervention was 
accomplishing that goal. 

First limitation: The Hawthorne effect. It 
was difficult to separate a Hawthorne effect 
from other effects due to specific aspects of the 
intervention. From the outset, all of the stake- 
holders in each of the three schools expressed 
unmitigated enthusiasm for the project, and 
they were clearly excited by the facts that their 
schools had been selected to participate in a 
project supported by a major federal grant. In 
addition, these schools, like many other schools 
in our experience, had a high commitment to 
integrating technology into instruction and they 
were seeking guidance to accomplish that goal. 
This commitment was heightened by a state- 
wide initiative to infuse technology into schools 
backed by substantial new funding from the 
state lottery earnings. 

School and central office administrators 
were clearly interested in capitalizing on the 
public relations value of the project as evi- 
denced by the facts that articles about the 
project appeared in local papers, the project 



was highlighted in applications for a state-wide 
“school of excellence” award, and members of 
the university research team were invited to 
make presentations about the project at meet- 
ings of the parent-teacher organization and the 
local school board. Parents expressed their 
support for the project in terms of their belief 
that a familiarity with technology was impor- 
tant to their children, and students were excited 
about being able to use the computers. The 
effects of this general enthusiasm and sup- 
port can be seen in the following transcript 
of tape recorded reactions one of us dictated 
after attending a meeting of the Parent 
Teacher Organization early in the school 
year: “Several parents commented to me 
how supportive they were of the project 
being at the school. One father said that his 
son had brought a book home in anticipation 
of the beginning of the project.” Also, 
teachers, perhaps because of the high enthu- 
siasm and expectations, suggested that they 
were apprehensive about the project’s suc- 
cess. One of the teachers recorded in her 
log: “I had my interview with Dr. [member 
of the university research team] today. I was 
very relieved to find that they [research 
team] were pleased with our progress. A lot 
of my apprehension was relieved.” 

The enthusiasm, high expectations, and 
apprehension created by the project clearly had 
potential to affect the pedagogical goal. In fact, 
as discussed earlier in this report, we believe 
these factors figure prominently in explaining 
the decline of morale at Collins School. How- 
ever, other considerations place the limitation 
implied by the Hawthorne effect in perspec- 
tive. First, we did collect data in several con- 
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trol classrooms that were using a computer pro- 
gram designed to promote independent reading 
by awarding points when students demonstrated 
adequate knowledge of a book’s content. In 
addition, the project acquired a lower profile as 
the year progressed and teachers became more 
accustomed to the idea of a formative experi- 
ment in which the intervention could be 
adapted in response to negative outcomes or 
problems. The fact that we worked in the 
participating schools for an entire school year 
also mitigates over time against the Hawthorne 
effect. Lastly, we believe that the Hawthorne 
effect is to some degree inextricably linked to 
interventions involving computer technology in 
schools, at least currently. That is, given the 
present interest in using technology in schools 
and the relative novelty and unconventionality 
of computer-based activities, the Hawthorne 
effect is not just a nuisance variable but one 
that merits study in its own right as a factor 
that may influence the effectiveness of com- 
puter-based interventions. 

Second limitation: Detectingsubtle changes 
in students’ independent reading. From the 
outset of this project, we believed that the 
effects of multimedia book reviews on students’ 
independent reading were likely to be less 
direct and less immediate than other interven- 
tions such as providing extrinsic rewards for 
the number of books read. However, our 
pedagogical theory values more highly those 
activities designed to develop long-term, intrin- 
sic motivation to read as included in what 
Alvermann and Guthrie (1993) describe as the 
engagement perspective. Given that the inter- 
vention proceeded from this perspective, we 
expected changes in students’ independent 



reading to be subtle and to emerge over an 
extended time. Likewise, we expected that 
detecting such Changes and connecting them 
unequivocally to the intervention would be 
difficult. Not only was it difficult to connect 
the intervention to observed movement toward 
the pedagogical goal, it was also difficult to 
determine specific aspects of the intervention’s 
implementation that enhanced or inhibited 
progress toward the goal. 

To address this limitation, we collected a 
variety of quantitative and qualitative data 
aimed at determining the amount and diversity 
of students ’ independent reading throughout the 
school year. Quantitative data included having 
students complete the baseline and postexperi- 
mental ERAS (McKenna & Kear, 1990) and a 
questionnaire concerning the diversity of their 
independent reading. Additionally, parents 
completed a questionnaire concerning students’ 
reading outside of school. These data are 
reported in Table 2. Qualitative data included 
observational field notes, semistructured and 
focused group interviews with teachers and 
students, videotapes, and teachers’ logs. None- 
theless, our difficulties in collecting data were 
evident in the following excerpt from our field 
notes. 

I worried that I was just not seeing 
what I was “supposed” to be seeing. I 
saw the good readers sneaking books to 
read under their desks and relying on 
themselves or close friends for book 
suggestions. I don’t know that with the 
time I spent in the classroom that I 
would ever see the students diversify 
their own reading or increase it. I was 
simply not there all the time, and I was 
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not their teacher. If I taught these stu- 
dents every day, I would see what was 
in their bookbags, I would hear their 
stories, I would see their interactions in 
the classrooms, I would see what is 
done with their free or extra moments. 

The short time segments that I spent in 
the classroom just didn’t let me see 
what needed to be seen. 

Consequently, we tried to make our class- 
room visits at times we thought students would 
have some free time to read. We also relied 
heavily on teachers’ comments during inter- 
views and in their logs to monitor the effects of 
the project on students’ independent reading. 

Third limitation: Baseline levels of inde- 
pendent reading. As noted in the section 
reporting baseline data, the majority of the 
students involved in this project already 
engaged in a good deal of independent reading 
and had positive attitudes about in-school and 
out-of-school reading. Under such conditions it 
is less likely that we would be able to deter- 
mine whether the multimedia book review 
activity was increasing independent reading 
than if there were many students who did little 
independent reading. Determining the effects of 
the intervention on independent reading was 
further complicated by the documented finding 
that independent reading tends to decrease as 
students advance through school (see Foertsch, 
1992; McKenna, Kear, & Ellsworth, 1995). 
From this point of view, maintaining students’ 
level of independent reading over an extended 
period of time can be seen as movement in a 
positive direction relative to this trend. Indeed, 
this phenomenon enters into our subsequent 



presentation and interpretation of the ERAS 
data. 

Findings from the quantitative data. Quan- 
titative data were obtained from (a) the ERAS 
(McKenna & Kear, 1990), (b) a questionnaire 
designed to measure the diversity of students’ 
reading, (c) a questionnaire designed to mea- 
sure parents’ perceptions of students’ reading 
outside of school, and (d) the Deford Theoreti- 
cal Orientation to Reading Profile (TORP; 
DeFord, 1985). Students completed the ERAS 
and the diversity questionnaire; their parents 
completed the parent’s questionnaire; and, 
teachers completed the TORP. Participants 
completed these instruments during the first 
month of the school year prior to beginning the 
intervention and again during the last few 
weeks before the end of the school year. 
Tables 1-4 show pre- and postexperimental 
results including the gain or loss by various 
categories. Also shown, are those pre- to post- 
experimental differences that are statistically 
significant when compared using a f-test for 
correlated samples. 

The results of the ERAS shown in Table 1 
indicate that for the intervention classes, statis- 
tically significant gains in mean raw scores 
were evident in one class on the recreational 
reading subscale, in two classes on the aca- 
demic reading subscale, and three classes on 
the total across both subscales. In the two 
comparison classes where a computer program 
was used to award points for reading books, 
statistically significant decreases in raw scores 
were evident in one of the classes on the recre- 
ational reading subscale and in both classes on 
the academic subscale and on the total scores 
across both subscales. Changes in scores on the 
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Table 4. Means (Standard Deviations) for Students’ Scores on the “Choosing Things to Read” Questionnaire 





Time of Administration 


School/Teacher 


Fall 


Spring 


Gain/Loss* 


Collins School 
Andrews 


56.00 


53.64 


(2.36) 




(12.64) 


(12.16) 




Broward 


61.61 


49.83 


(11.78)* 




(13.09) 


(11.05) 




Hartwig School 
Burton 


52.55 


44.45 


(8.09)* 




(14.43) 


(14.89) 




Palmer 


31.27 


44.91 


13.64** 




(11.88) 


(10.68) 




Pearson 


54.35 


51.39 


(2.96) 




(17.49) 


(17.57) 




Borders School 
Morris 


52.51 


56.00 


3.79 




(13.40) 


(18.45) 




Sievers 


47.33 


53.81 


6.48 




(14.45) 


(16.41) 




Comparison Classes 
Teacher 1 


51.25 


52.00 


.75 




(1151) 


(13.86) 




Teacher 2 


58.00 


46.68 


(11.32) 




(12.47) 


( 9.27) 





a a = gain(loss) 

> < .05. "p < .001. 

ERAS from the beginning to the end of the 
school year must be interpreted in light of the 
tendency of elementary school students’ raw 
scores to decrease over time (Foertsch, 1992; 
McKenna et al., 1995). For example, a raw 
score of 52 on the ERAS results in percentiles 
of 35, 42, and 49 for fourth-, fifth-, and sixth- 
grade students, respectively. These data sug- 



gest that students attitudes toward academic 
and recreational reading tend to increase (or at 
least not to decrease at expected levels) more 
in the classes involved in the multimedia book 
review activities than in two classes using an 
alternative computer-based activity aimed at 
increasing independent reading. 

Means and standard deviations by class for 
the “Choosing Things to Read” questionnaire 
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are shown in Table 4 (see Appendix B for the 
complete questionnaire). Pre- and postexperi- 
mental means for four classes involved in the 
intervention decreased, two of which were 
statistically significant, and means for three 
classes increased, one of which was statistically 
significant. The mean for one of the compari- 
son classes increased and one decreased; nei- 
ther of which was statistically significant. 
These data do not clearly support or contradict 
an assertion that the intervention had any effect 
on the diversity of children’s reading across 
schools. 

Means and standard deviations by class on 
selected variables from the Parents’ question- 
naire are shown in Table 2 (see Appendix C for 
the complete questionnaire and Figure 11 for 
an explanation of each variable). Among the 
seven classes involved in creating multimedia 
book reviews, statistically significant gains on 
variables included on the parents ’ questionnaire 
were as follows: free time reading (1 class); 
ratio of time spent watching TV and reading (2 
classes); number of children’s books in the 
home (1 class); range of reading materials (1 
class); number of trips to the library (1 class); 
possession of a library card (1 class); and 
reading for enjoyment at home (2 classes). 
None of the decreases in means were statisti- 
cally significant. Among the seven classes 
involved in the study over 2 years, one class 
had 4 statistically significant increases (Ms. 
Palmer at Hartwig), two classes had 2 statisti- 
cally significant increases (Ms. Broward at 
Collins School and Ms. Sievers at Borders 
School), three classes had 1 statistically signifi- 
cant increase (Ms. Andrews at Collins School, 
Ms. Burton at Hartwig, and Ms. Morris at 



Borders School). No statistically significant 
increases or decreases on the variables identi- 
fied in the Parents’ questionnaire were ob- 
served in Ms. Pearson’s class at Hartwig 
School or in either of the comparison classes. 
However, the statistical significance of these 
findings must be interpreted in light of the 
many comparisons conducted across class- 
rooms and variables. 

Findings from qualitative data. Teachers 
and parents clearly observed increases in some 
students’ independent reading, which they 
connected to the project activities. Their com- 
ments in interviews, audiotaped and videotaped 
meetings, project logs, and off-hand comments 
recorded in our field notes, consistently refer 
to positive changes in students’ independent 
reading, which was manifested in various 
ways. 

Our data contain frequent reports of chan- 
ges in individual student’s reading as reported 
by parents and teachers. For example, our field 
notes from Hartwig school include the follow- 
ing notation: 

I was talking [with a parent who helped in 
the lab] and she said she didn’t know if it 
was due to his project, but she had noticed 
a very significant, noticeable improvement 
... in her daughter’s reading at home. 

She had said that her daughter has many 
books at home before and would occasion- 
ally read parts of them but would never 
seem to finish them . . . whereas now she 
observed her daughter doing much more 
reading, finishing the books and talking 
more about them. . . . 
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Likewise, Ms. Pearson recorded the following 

in her log: 

Several parents told me throughout the 
year how pleased they were that we were 
involved in this research project. They 
said their children had always had plenty 
of books at home, but they never seemed 
to completely read their books. They (the 
parents) were able to see drastic changes 
in their children throughout the year. 
Their children were now completely read- 
ing books and asking for more. The par- 
ents were very excited about the changes. 

And, on another occasion she wrote again in 

her log: 

I tutor a third grader. One afternoon when 
I took him home from tutoring, his 
mother wanted to know if we would be 
doing the research project with the 
computers when [her younger son] got 
to 5th grade. She wanted us to because 
Karen, her daughter in Ms. Palmer’s 
class, had greatly benefitted from this 
year’s project. She said that Karen was 
reading at home all the time. She also said 
that Karen’s writing skills had greatly 
improved this year due to using comput- 
ers to do book reports. 

Ms. Burton wrote in her log in February, 

Today I noticed that Candace had a col- 
lection of books on her desk. I asked her 
if she was reading now and did she enjoy 
it. She showed enthusiasm about reading 
and told me about her books. I asked her 
if she felt the computer book reports had 



aided her in choosing to read and she said 
she liked putting her work on the com- 
puter. 

During a visit to observe in Ms. Broward’s 
class at Collins School, she pointed out that 
Mitch, a poor reader, was unexpectedly bring- 
ing abridged classics to school to read. We 
discovered that he was asking his mother to 
buy these books when he accompanied her to 
the supermarket. He was anxious to show us 
his new book each week and explained that he 
wanted to get them to enter into the computer. 

Some of the teachers observed positive 
changes across all of the students in their 
classes, which they attributed to the project 
activities. For example, Ms. Burton wrote in 
her log, “I saw a lot of growth in my class in 
many ways. Toward the end of school, I saw 
kids through with work reading books. I saw 
them completing work in order to read their 
book.” As discussed in a previous section, 
other teachers observed that the effects were 
more obvious with readers of high or low 
ability. For example, Ms. Pearson stated in an 
interview that she thought the project “encour- 
ages students to read, especially those with 
lower reading abilities. ...” Ms. Ellers, on 
the other hand, saw more of an effect on the 
better readers as evidenced from her response 
when we asked her what she saw happening 
relative to the pedagogical goal: “With the top 
readers, yes, this [project] is a hit. With the 
high ability/high interest readers it amplifies 
that effect. ” 

Teachers frequently observed connections 
between project activities and classroom events 
that they perceived as positive changes related 
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to the amount of children’s reading. For exam- 
ple, Ms. Pearson stated in a videotaped inter- 
view, 



that he could do it on that one. To actually 
take the time to get it and go and do it— it 
was a real surprise; a pleasing surprise. 



Today, two low-average readers turned in 
Troll book orders. What impressed me 
most was that these 2 students ordered 
Troll [book club] at home for the summer. 
It is now February. In 13 years of teach- 
ing 5th grade, I have never had any stu- 
dents to order Troll at home even closer to 
the summer time. 



Ms. Palmer recorded in her journal: “I’m 
noticing more students are asking for additional 
pages to record books. ” Ms. Sievers at Borders 
School related the following incident during an 
interview. 



I know we finished reading the book 
Summer of the Monkeys and I was real 
surprised— last week I guess it was, one of 
the students, Paul, came up to me and 
asked if he could borrow it. It was one 
that was real special to me. I didn’t know 
what he was going to do with it at that 
time. I said why are you going to borrow 
it when we’ve already read it, and he said 
he was going to do HyperCard, do it on 
the book report, so he took it with him. 
Then the next day— the next time when he 
came back and asked me if he could bor- 
row it again and he could not find it, he 
got very disturbed that he couldn’t find it. 
So he said maybe Andy had apparently 
gotten it to go and do his book report on it 
also. It is a thick book— a fairly hard book 
just to read by yourself. But that is some- 
thing that Andy, not being a good reader— 
for him to take the initiative and either do 
it on his own or have someone tell him 



The qualitative data that we gathered 
provides little evidence of an increase in the 
diversity of students’ independent reading, at 
least in terms of books. Teachers rarely made 
comments related to diversity in their logs or 
brought up this aspect of the pedagogical goal. 
In fact, when asked directly about the diversity 
of students’ reading, the teachers indicated 
that they were not seeing any positive or 
negative changes. For example, Ms. Ellers 
responded as follows: 

Now as far as diversity, I’m not seeing a 
whole lot of diversity because they’re 
[into] series books and you know some 
little girls like the Sweet Valley Twins or 
whatever . . . they’re coming in here with 
a notebook or sheets of papers, so amount 
I’ve seen more, but not diversity . 

In the same discussion, Ms. Sievers agreed, 
adding M I’m seeing not much diversity also 
(either). They kind of read the same things: 
fourth-grade books, chapter books.” 

Nonetheless, teachers occasionally saw 
changes that might have indirect effects on 
diversity. For example, Ms. Palmer commented 
in her log that as a result of the project, 
students were beginning to realize “that this 
[activity] was not about book size (number of 
pages)— but to read whatever interests them 
and use this reading to relate or pass on to 
other students their opinions of various litera- 
ture.” We also observed that focal students 
read diverse materials. For example, we re- 
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corded the following in our field notes: “Eliza- 
beth (hi ability /hi interest) was observed having 
different paperback books in her desk on three 
occasions.” Likewise, when we received per- 
mission to look at what books were in students’ 
desks and when we observed them during free 
reading times during February, we saw evi- 
dence that many individual students were 
reading diverse materials such as nonfiction, 
fantasy, and football souvenir programs. How- 
ever, this diversity did not seem to be con- 
nected directly to the multimedia book reviews 
or influenced greatly by this activity. Effects in 
this area seemed to be indirect at best, as 
proposed in a previous section discussing how 
dealing with some of the technological compo- 
nents of the activity had secondary effects on 
the amount and diversity of students’ reading. 

Several possible explanations may account 
for this finding. First, teachers seemed to relate 
more directly to the goal of increasing indepen- 
dent reading than to increasing diversity. This 
orientation may have subtlety affected the way 
they implemented or reinforced project activi- 
ties and/or selectively influenced their percep- 
tions. Also, the project activities focused on 
books, not other reading materials such as 
magazines, which may have given an overly 
narrow view of students’ reading. Likewise, 
because the database activity was not fully 
implemented, students did not participate for an 
extended period of time in a part of the project 
with greater potential to increase diversity. 

Students ’ Increased Concern for Writing 

Although the teachers were aware that we 
were most interested in their observations 



pertaining to the pedagogical goal of increasing 
independent reading, they were also encour- 
aged to note other effects of the project activi- 
ties. In this regard, they often focused on the 
benefits of the program in helping students 
become more proficient in the technical aspects 
of writing and of editing. Students increased 
attention to technical aspects of their writing 
were not only quite visible but also seemed to 
be especially noteworthy in teachers’ explana- 
tions for the value of the activity. In their logs, 
teachers repeatedly made comments such as the 
following. 

• ... it seems the kids are being more careful 
about word usage, spelling, and punctuation. 

• It is interesting to watch how much easier the 
kids are catching mistakes (spelling & usage) 
of each other. There is a lot of revising going 
on when these kids pair up to create their 
stacks. 

• Their essays [in a writing contest entered by all 
of the fifth-grade students] were much im- 
proved over prior students. We feel like this 
project has helped students’ writing skills as 
well as reading skills. 

• We are working in [the lab] reviewing one 
another’s work— editing, etc. Amazing how 
students can see mistakes of other students and 
not recognize their own. 

• I do feel that our students’ exposure in com- 
puter program helped them to be better editors. 

• ... this experience has made them [students] 
aware of the writing/proof reading/editing 
process. 
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Clearly, the involvement in creating 
multimedia book reviews had an effect on 
students’ writing as we also noted indepen- 
dently in our field notes in comments such 
as the following. 



I’ve noticed . . . that students are much 
more careful with their writing. They 
ask how to spell words when writing. 
[Ms. Burton] notice too that students 
working in the lab are more concerned 
about their spelling. [She felt that] 
students feel some ownership over their 
book reviews. 



This increased care in writing seemed 
related to students viewing the multimedia 
book reviews as public documents intended for 
use by teachers, parents, the university re- 
search team, and other students. Jason, one of 
the focal students at Hartwig school, stated, “I 
have to fix that [spelling error] because I don’t 
want anybody to think I’m dumb.” Students 
also seemed to gain satisfaction from finding 
errors in spelling, grammar, and punctuation 
on those occasions when they read each others’ 
reviews. Their attention to others’ work may 
have increased attention to their own, perhaps 
to avoid criticism from their peers. Teachers 
also emphasized technical correctness in stu- 
dents’ reviews, which may have heightened 
students’ concern for technical correctness. 
Several of the teachers insisted that students 
write out a technically correct version of the 
text of their reviews before they were allowed 
to enter it on the computer. However, for the 
most part they did so in an off-hand, informal 
way rather than badgering students about their 
writing. 



Miscellaneous Preliminary Findings 
and Unanticipated Effects 

We report here several preliminary find- 
ings and observations that may be of interest to 
those who wish to conduct related formative 
experiments. These preliminary findings were 
not central to our stated pedagogical goal nor 
to our interest in determining the degree to 
which computer-based activities were fully 
appropriated by teachers and students. Neither 
are these findings based on extensive, system- 
atic data collection and analysis. 

1. Typing was not a major obstacle for 
students in this activity. Although none of the 
students had highly developed typing skills, 
they were content to type slowly when entering 
textual information. Students virtually never 
complained about having to type information, 
despite that it was a laborious task for most of 
them. Moreover, students working in pairs at 
a computer were usually content to wait on 
their partners when typing information. 

2. Participation in this project had a 
notable effect on teachers’ professional 
development that extended beyond the project. 
Although this project did not have all of the 
marks of a true collaborative research project 
between classroom teachers and university 
researchers (see Allen et al., 1992; Anders, 
1996; Jervis et al., 1996), teachers seemed to 
benefit professionally from their involvement 
in the project. All of the teachers presented at 
least once at a professional conference (to 
our knowledge, none had done so before); 
one teacher began an advanced degree pro- 
gram in conjunction with the project while 
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another incorporated it into an on-going pro- 
gram. Most notably, three teachers submitted 
proposals to a state conference on their own in 
the year following the completion of the pro- 
ject. Several of the teachers noted in their logs 
that these activities were somewhat intimidating 
but at the same time rewarding and profession- 
ally meaningful. 

3. Parental involvement in the classroom 
and school increased in Collins and Hartwig 
Schools where a concerted attempt was made to 
recruit and train parents to assist with the 
HyperCard programming. Increased parental 
involvement was also a by-product of the video 
created by Ms. Pearson’s class because parents 
were enlisted to make costumes, find props, 
and donate video equipment. Even in Borders 
School where parents were not recruited to 
assist with the day-to-day multimedia book 
review activities, many parents attended a 
school technology fair where the multimedia 
book review project was among the displays. 
Many parents seem to have an inherent interest 
in technology on behalf of their children and 
enthusiastically support such efforts. Their 
involvement seemed to be valued equally by 
teachers, administrators, and students, although 
for reasons that are likely to be somewhat dif- 
ferent for each group. Parents seemed to react 
positively to participating in learning about 
technology in a nonthreatening way with teach- 
ers and their children. Our experience in this 
project fits well within and reinforces the 
model for creating a community of learners 
through the implementation of computers as 
proposed by Keeler and Alexander (1994). 

4. The project clearly increased the ability 
of teachers and students to use the computer 



effectively. They learned specific skills related 
to HyperCard such as being able to create 
buttons and to link cards, which can be trans- 
ferred to other applications. They also learned 
general information about hardware and soft- 
ware such as how to hook up and use an LCD 
panel or how to move files from a disk to the 
hard drive. Also as a result of their participa- 
tion in the project, they learned how texts 
might be incorporated with other media to 
create electronic documents . Long-term experi- 
ences with such activities have demonstrated 
that students change their conceptions of writ- 
ing and reading (see Tierney et al., 1992) 

Discussion 

The primary purpose of the present study 
was to investigate how a multimedia book 
review activity might be implemented in mid- 
dle-grade classrooms to effect increases in the 
amount and diversity of students’ independent 
reading. That purpose was derived from our 
interest in an approach to classroom research 
that Newman (1990) has described as a forma- 
tive experiment. Formative experiments ad- 
dress a domain of questions and issues that are 
particularly relevant to the use of computer 
technology in schools and that have not been 
addressed by conventional experiments, nor 
typically by qualitative approaches to research. 
A secondary purpose was to expand and to 
refine our knowledge about the concept of a 
formative experiment as a an approach to 
classroom research and about the practicality of 
conducting such research. In this section, we 
discuss each of these purposes separately. 
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Discussion of Results 

Both the quantitative and qualitative data 
indicate that the multimedia book review activ- 
ity contributed to achieving the pedagogical 
goal of increasing the amount of children’s 
reading. Although, as one might expect to 
discover using a formative experiment, the 
manner in which the activity contributed to the 
pedagogical goal was not necessarily antici- 
pated at the outset of the project. The interven- 
tion’s effect on achieving the pedagogical goal 
was clearly mediated by students’ and teachers’ 
responses to the challenges of working with the 
technology, particularly HyperCard, not pri- 
marily by the creation of multimedia book 
reviews and sharing them with others as antici- 
pated. The introduction of the multimedia book 
review activities represented a novel intrusion 
into normal classroom routines, which was 
greeted with much enthusiasm by teachers and 
students as well as administrators and parents. 
That climate combined with a change in the 
social dynamics of instruction and the increased 
engagement of students and teachers led to 
increased interactions about and enthusiasm for 
books, which in turn led to more independent 
reading. Thus, the increase in independent 
ready was more a by-product of students’ and 
teachers’ primary focus on mastering the 
technology, particularly HyperCard. This 
conclusion is reflected in the following tran- 
scription from one of our tape-recorded field 
notes at Hartwig School during February: 

[Ms. Pearson’s comment] leads me to 

think that maybe some of the effects of the 

project are indirect in the sense that teach- 



ers’ involvement with the project are 
leading them to emphasize independent 
reading more which filters down to the 
students so that some of the effects we’re 
seeing maybe during the project are not 
directly due to the students entering book 
reviews but the total impact that the proj- 
ect is having on the school or the teachers 
such that independent reading is empha- 
sized more. 

Put another way, our pedagogical goal of 
increasing the amount of independent reading 
was furthered more by connecting books to an 
engaging, challenging use of the computer than 
by the capability of the technology to encour- 
age sharing of information about books. This 
realization gradually led us to see our invest- 
ment of considerable time and effort into 
having teachers and students learn HyperCard 
not has a frustrating distraction from our in- 
tended goal but as an important way to enhance 
it. Thus, in the project’s second year, we felt 
justified in refining the activities designed for 
teachers to learn HyperCard who would then 
teach it to students. 

We believe this finding to be important 
because it could be argued that the intrigue of 
the technology would be so distracting as to 
work against the pedagogical goal, which 
would call into question the utility of the entire 
activity for accomplishing the identified goal. 
Also, we believe the fact that involvement with 
technology tended to mediate the accomplish- 
ment of a valued pedagogical goal is consistent 
with the perception of many educators that 
technology can be valued intrinsically for that 
purpose. It is not trivial to determine that 
technology can act as a catalyst for changing 
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the typical interaction patterns among teach- 
ers and students. Such changes are the basis 
for arguments that technology offers strong 
potential for bringing about school reform 
(see Means, 1994). Thus, the results of this 
study have implications that extend beyond 
the particular pedagogical goal and the 
instructional intervention investigated. Not 
only may the use of technology in schools 
positively alter social interaction patterns in 
schools, it may further curricular goals in 
various subject areas by embedding relevant 
content within challenging and engaging 
computer-based tasks such as using Hyper- 
Card. In that regard, the findings of the 
current investigation are consistent with 
other studies indicating that challenging uses 
of technology can over time further the goals 
of literacy instruction in schools (e.g., 
Tierney et al., 1992; Turner & Dipinto, 
1992). 

On the other hand, we found that the extent 
to which the technology was fully appropriated 
by teachers was decidedly mixed. Factors that 
seem to account for differences include the role 
that teachers assumed in relation to the activi- 
ties, the logistical arrangements that were 
necessary to carry out the activities, and the 
degree to which the overall climate of a school 
and classroom led teachers to be flexible and 
independent. Although there was considerable 
variation from classroom to classroom, we 
found that teachers readily accommodated the 
multimedia book review activities into their 
instructional program, but the level of integra- 
tion into their teaching beyond the project 
activities tended to be minimal and superficial, 



especially in the early stages of the project. For 
example, in the early stages of the project, 
integration rarely extended beyond occasionally 
rescheduling previously-planned activities to 
work extra time on multimedia book review 
activities. As the year progressed in several of 
the classrooms, there was some evidence that 
project activities were being extended and 
integrated into other classroom activities, most 
notably in Ms. Pearson’s class with increased 
word processing activities and her implementa- 
tion of the group book review activity related 
to her social studies unit. This minimal exten- 
sion and integration coupled with the teachers’ 
connection of the project activities with con- 
ventional curricular goals (e.g., technical 
aspects of writing) are consistent with much 
previous research findings on the effects of 
introducing technology into classrooms (see 
Bruce & Rubin, 1993; Miller & Olson, 1994). 
It is also consistent with a defining attribute of 
formative experiments as described by New- 
man (1990) who argues that many endpoints 
are possible in a particular experiment and that 
“ [t]he environment may transcend its initial 
goals. It may also retain goals and organization 
in spite of the technology designer’s concerted 
efforts to support alternative models” (p. 10). 

Thus, although computer-based activities 
such as the one in this study seem to have 
strong potential to alter positively the social 
dynamics between teachers and students, this 
effect in itself does not seem to be adequate 
to transcend teachers’ commitment to con- 
ventional instructional goals and activities. 
It seems to us, however, that teachers may 
be much more receptive to transforming 
their instruction within the context created 
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by a computer-based activity that points in 
that direction, although the data in this 
investigation only support that contention 
indirectly. One development that does sup- 
port such an interpretation is that the project 
activities continued to be implemented in 
varying degrees at Hartwig and Borders 
Schools into a subsequent school year despite 
the fact that the project had concluded in both 
schools. 

We found little evidence that the activity, 
as implemented, had an effect on the goal of 
increasing the diversity of students’ indepen- 
dent reading. One explanation for the lack of 
results in this area is that we did not fully 
implement the database activity, which was 
designed specifically to expand students’ aware- 
ness of a variety of books being read by their 
classmates. Another factor may be that teachers 
seemed less conscious of and concerned about 
this goal. It is also possible that the multimedia 
book review activity may not be adequate in 
itself to overcome students’ propensity to read 
a narrow range of topics, genres, and authors. 
Broadening students reading may depend upon 
coordinating and supplementing the multimedia 
book review activity with other classroom 
activities in the areas such as literature re- 
sponse, thematic units, and reading diverse 
texts aloud to students. 

Not surprisingly, we found that the school 
environments and teachers’ roles to some 
extent shaped the effects of the multimedia 
book review activity. For example, the positive 
effects of the project seemed enhanced by a 
supportive, nonthreatening school environment 
in which teachers felt supported and trusted to 
make their own decisions and had the freedom 



to deviate from set schedules and established 
curriculum. The availability of equipment such 
as an LCD panel for group presentations 
changed considerably not only the options for 
implementing the activities but the type of 
social interactions that occurred during the 
activities. Given our experience, we are con- 
vinced that it would be difficult, if not impossi- 
ble, to carry out the multimedia book review 
project without at least one teacher who as- 
sumed the role of “technology expert” as 
discussed in the results section of this report. 
These differences had to be taken into account 
as we assisted teachers who were to implement 
the project in each school, and which led to 
variations within the overall framework of 
the multimedia book review activity. How- 
ever, when using a formative experiment as 
an approach to classroom research, these 
differences become opportunities to extend 
understanding of the relation between the 
intervention and the pedagogical goal. 

Through the use of a formative experi- 
ment, we were also attuned to unanticipated 
outcomes of the intervention not directly re- 
lated to our pedagogical goal but that might 
provide the basis of further study. That is, it is 
possible that the same or a similar intervention 
might be studied in relation to other pedagogi- 
cal goals. For example, the multimedia book 
review activity as carried out in this investiga- 
tion seemed to have an effect on students’ 
writing and on teachers’ perceptions of their 
writing. Likewise, other outcomes that might 
be investigated include increased parental 
involvement, increases in computer skills and 
uses, attitudes toward computers, and teachers’ 
professional development. 
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Discussion of the Present Investigation 
as a Formative Experiment 

The concept of a formative experiment 
guided this investigation. To our knowledge, 
the explication of what a formative experi- 
ment is and how it might be conducted is 
limited to (a) a single article by Newman 
(1990) who proposed the concept and pro- 
vided a single example of a study from his 
own work, and (b) our own speculations 
about its utility as a form of classroom 
research and about a framework for designing 
and conducting such research (see Reinking & 
Pickle, 1993; Baumann et al., 1996). Thus, it 
seems warranted that we comment briefly here 
on how our experience in using this approach 
compared to our abstract musings about its 
potential merit. 

Our most poignant discovery is that the 
formative cycle of data collection and resulting 
adjustments to the intervention must be consid- 
ered fluid as opposed to a discreet and orderly 
progression of events. An advantage of ap- 
proaching classroom research from the stand- 
point of a formative experiment is that the 
effects of instructional interventions are ac- 
knowledged to be the product of a complex 
array of interacting variables within the school 
and classroom environment. We believed at the 
outset of our investigation that through the 
application of rigorous research methods, we 
could isolate these variables and their role in 
enhancing or inhibiting the intervention’s 
effectiveness toward accomplishing the peda- 
gogical goal. Moreover, we thought that when 
a single variable was identified, the interven- 
tion could be modified at a definite point in 



time in response to it. We discovered that the 
multiple levels on which formative adjustments 
needed to be made almost on a daily basis did 
not permit the detached reflection and carefully 
reasoned decisions that we had hoped. Retro- 
spectively, we do think, however, that future 
formative experiments might consider develop- 
ing and using a process/product matrix that 
Patton (1990) has proposed as useful in evalu- 
ating programs. General components of that 
matrix might include concerns in the following 
areas: logistical (in terms of implementing the 
intervention), methodological (i.e., modes of 
data collection), pedagogical, technological, 
interpersonal, and ethical. Such a matrix may 
not eliminate the need for more fluid decision 
making, but may aid in documenting more 
clearly the relationship between variables and 
adjustments that were made in response to 
them. 

Another issue we were able to address in 
our exploration of using a formative experi- 
ment was how to write a report of our investi- 
gation. We found that it was possible to use the 
conventional structure reporting a quantitative 
study to be suitable for reporting our formative 
experiment. That is, we began with a rationale 
for our investigation, including a literature 
review. However, unlike the introduction to a 
typical research report, we also included our 
pedagogical goal, the pedagogical theory that 
justified it, and a description of the instruc- 
tional intervention we believed addressed our 
goal. The remainder of the report followed 
convention closely with sections pertaining to 
method, results, and a discussion of findings. 
Although we are not convinced that this is the 
best or only reasonable way to report formative 
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experiments, to us it seems advantageous that 
a familiar structure works. 

The concept of a formative experiment 
needs more careful scrutiny and thought before 
it can be clearly articulated as a genre of class- 
room research. If it is to be widely accepted 
and used, it will need not only a coherent 
rationale, but also general guidelines for con- 
ceptualizing investigations; for planning their 
implementation; for gathering, analyzing, and 
interpreting data; and for reporting results. It 
will need a well-articulated epistemological 
rationale as well. We suspect that such a ration- 
ale may be found within pragmatic points of 
view as expressed by writers such as Cherry- 
holmes (1993). 

Despite the need for more elaboration and 
examples of formative experiments, we believe 
through our experience that they have strong 
potential as a useful, systematic alternative to 
existing approaches to classroom research. 
Among the advantages of formative experi- 
ments is that they more readily acknowledge 
and address the inherent complexities of class- 
rooms while seeking ways to address that 
complexity toward the attainment of well- 
defined pedagogical goals. As such, they 
reflect, albeit more systematically, the often 
trial-and-error aspects of good teaching. This 
similarity to teaching may increase the recep- 
tiveness of classroom teachers to the research 
enterprise and to defining themselves as an 
integral part of that enterprise as teacher/ 
researchers. This perspective may also appeal 
to educational policy makers and parents who 
can relate to the fact that formative experiments 
involve clear-cut pedagogical goals while at the 
same time impressing upon them that rarely are 



there unqualified instructional recommenda- 
tions that can assure the accomplishments of 
those goals. Nonetheless, formative experi- 
ments do not exclude the possibility of general- 
izations. Factors affecting movement toward 
the pedagogical goal may be found to cut 
across different even distinctly different envi- 
ronments that could lead to general recom- 
mendations. 

In the end, we agree with Eisenhart and 
Borko (1993) that useful classroom research 
must transcend any particular paradigm and 
that researchers need to be creative and eclectic 
in their approach to investigating teaching and 
learning in schools. Based on our experience, 
we think that formative experiments have 
strong potential as a framework for providing 
such flexibility and eclecticism while conform- 
ing to the principles they propose should guide 
any form of classroom research. In that regard, 
we believe formative experiments merit 
attention by a broad spectrum of classroom 
researchers, not just those interested in 
technology. Thus, we hope that this investiga- 
tion will generate interest not only in our 
findings relative to using multimedia book 
reviews to enhance the amount and diversity of 
independent reading, but also in the concept of 
a formative experiment as a form of classroom 
research. 
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Appendix A 

Semistructured Interview with Teachers Before Project Began 

Experience With and Attitudes About Computers 
Structured Interview With Project Teachers 



1 . What was your experience with computers prior to this point? (Outside of school/in school) 

• Do you own a computer? 

• Would you buy one if you had the resources? 

• What would you use the computer for? 

2. What were your attitudes toward computers prior to this project? What are your attitudes 
toward computers at this point in time? 

3 . Have you had any opportunities to observe students using computers? If so, could you describe 
the situations in which you have observed them? What were some of your observations? 

• What do you think are the students’ view of computers? 

4. How do you think other teachers in general view computers? The teachers in your school? 
Administrators? Parents? 

5. What positive outcomes do you hope for in this project? What negative outcomes are you 
concerned with? 

6. Imagine you are in a school that has as many computers as there are teachers. If you were 
responsible for making the decision about whether the computers would all be put in a lab or 
one in each classroom what would you do? 

7. Can you remember when you first thought about the possibility of using a computer for 
teaching? Describe what you remember. 

8. Has the computer affected your teaching in any way? 
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Appendix B 

Questionnaire on Diversity of Reading Completed by Students 
Before the Project Began and at the End of the School Year 



CHOOSING THINGS TO READ 



Name 

Teacher 

Date 



Part One 

Directions: Suppose that your teacher says you must choose a book to read from the library. 
Which kinds of books might you choose to take home to read? Check any kind of book you might 
want to read. But, don’t check any you probably wouldn’t want to read. 



a book about how to play a sport 
a story that is a mystery 
a story about a pet 
a book about cars or trucks 
a science fiction story 
a story about Indians 
a story about kids my own age 
a book of fairy tales or myths 
a book on science experiments 
a book about animals 
a book about planes 
a book about dinosaurs 
a book of science experiments 
a book about how to cook something 
a book about trains 
a story about strange creatures 
a story about an adventure 
a book about how to take care of animals 



a book about an event in history 
a make-believe story 
an encyclopedia 

a book about a person in history 
a book about a movie or TV star 
a book of beautiful poems 
a story about people in other 
countries 

a story about scary things 
a story about a horse 
a story that makes me laugh 
a book of jokes 

a book of funny rhymes and riddles 
a book of cartoons 
a story about someone in a war 
a story that takes place in the 
future 

a book about volcanoes 
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a story about someone my age 
a book with maps 
a story about faraway places 
a story about monsters or 
strange creatures 
a book about things that are 
strange but true 



a story about boys and girls getting 
along 

a story about someone who likes to 
play sports 

a book about what I would like to 
be when I grow up 



What are some other kinds of books that you like to read? 



What are your favorite kinds of books? 



Part Two 

When you pick a book to read, what do you think about? 



Directions: Answer each question by putting an X in one of the boxes. 

1 . When you pick a book, how often do you think about how the cover of the book looks? 





once in 






never 


a while 


often 


always 



2. When you pick a book, how often do you think about how many pictures the book has? 





once in 






never 


a while 


often 


always 
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3. When you pick a book, how often do you think about how long the book is? 





once in 






never 


a while 


often 


always 



4. When you pick a book, how often do you think about what your friends say about the book? 





once in 






never 


a while 


often 


always 



5. When you pick a book, how often do you think about what your teacher says about the book? 





once in 






never 


a while 


often 


always 



6. When you pick a book, how often do you think about who the author of the book is? 





once in 






never 


a while 


often 


always 



7. When you pick a book, how often do you think about whether you will like the characters or 
not? 





once in 






never 


a while 


often 


always 



8. When you pick a book, how often do you think about whether you will learn something from 
the book? 





once in 






never 


a while 


often 


always 



NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55 



A Formative Experiment 



81 



Part Three 

Directions: Answer each question by putting an X in one of the boxes. 

1 . When you read on your own, how often do you read books? 





once in 






never 


a while 


often 


always 



2. When you read on your own, how often do you read magazines? 





once in 






never 


a while 


often 


always 



3. When you read on your own, how often do you read newspapers? 





once in 






never 


a while 


often 


always 



4. When you read on your own, how often do you look for information in books like an 
encyclopedia, dictionary, atlas? 





once in 






never 


a while 


often 


always 



5 . When you read on your own, how often do you read stories that have characters made up by 
the author? 





once in 






never 


a while 


often 


always 
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6. When you read on your own, how often do you read books that give information? 





once in 






never 


a while 


often 


always 



7. When your read on your own, how often do you read about how to do something? 





once in 






never 


a while 


often 


always 



8. When you read on your own, how often do you read about people who really lived and things 
that really happened? 





once in 






never 


a while 


often 


always 



9. When you read on your own, how often do you read books that are funny? 





once in 






never 


a while 


often 


always 



10. When you read on your own, how often do you not finish a book? 





once in 






never 


a while 


often 


always 



11. When you read on your own, how often do you read poetry? 





once in 






never 


a while 


often 


always 
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12. When you read on your own, how often do you read about sports? 





once in 






never 


a while 


often 


always 



13. When you read on your own, how often do you read about animals? 





once in 






never 


a while 


often 


always 



14. When you read on your own, how often do you read about science? 





once in 






never 


a while 


often 


always 



15. When you read on your own, how often do you read about adventure? 





once in 






never 


a while 


often 


always 



16. When you read on your own, how often do you read stories that are make believe? 





once in 






never 


a while 


often 


always 
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Appendix C 

Parent Questionnaire 

Completed Before Project Began and at the End of the School Year 



Parent Questionnaire 

We appreciate your cooperation in taking 10-15 minutes to complete this questionnaire, which 
should be returned to your child’s teacher. This information is important to participation in the 
special computer project. If you have your child deliver the completed questionnaire to her/his 
teacher, you may want to seal it in the attached envelope. Thank you for your assistance. 



Your child’s name: 



1. Of the following activities, check the three that your child is most likely to do during free 
time at home. 

talk on the telephone 
listen to music 
play video games 
watch TV 
play with toys 

other (describe) 



2. Circle the activity in number one that your child does most during free time. 

3. On average, how long does your child spend reading for enjoyment each day? 

hours minutes 

4. On average, how long does your child spend watching TV each day? 

hours minutes 



play outside 
work on arts and crafts 
read a book or magazine 
work on a hobby 
play a musical instrument 
dramatic or pretend play 
(skits, playing house 
outerspace, etc.) 
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5. Of the following statements, check the one that best applies to your child: 

My child rarely, if ever, reads anything for enjoyment. 

My child reads for enjoyment once in a while. 

My child reads regularly for enjoyment but not a lot compared to other activities. 
My child reads regularly and often for enjoyment. 

My child reads for enjoyment almost all the time he/she has an opportunity to do so. 

6. How would you rate your child’s reading ability? ( Check one.) 



Well above average 
Above average 
Average 



Below average 
Well below average 
I’m not sure 



7. Are there children’s books in your home for your child to read? yes no 

If so, about how many? ( Circle one.) 



8 . 

9. 



fewer than 5 5-20 20-50 50-100 more than 100 

Does your child use a computer at home? yes no 

If yes, what kind? 

Rate on a five-point scale how often your child does the following? ( Circle a number for 
each statement.) 

Never Very Often 



Order books from book clubs. 

Look at books/magazines in stores. 

Ask for books/magazines as gifts. 

Complain about having to read a book for school. 

Bring home something from school to read for enjoyment. 
Talk about something he/she has read. 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
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Never Very Often 



Ask to go to the library to find something to read. 1 

Look up information in the dictionary or encyclopedia. 1 

Read something to you. 1 

Read something in the newspaper. 1 

Give someone reading material as a gift. 1 

Read to another child. 1 

Say he/she doesn’t like reading. 1 

Recognize or talk about a particular author. 1 



Go to movies or watch TV programs about books he/she reads. 1 



Write her/his own “books” or stories for enjoyment. 1 

Read about places he/she will visit/is visiting on a trip. 1 

Say that reading is boring. 1 

Read cereal boxes or other materials while eating. 1 

Read in front of the TV. 1 

Choose Halloween costumes based on book characters. 1 

Use their own money to buy a book. 1 



2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 



10. Have you observed other behaviors (positive or negative, like the ones above) that indicate 
how much your child reads and how he/she feels about reading? If so, please describe them. 
(Use the back of this questionnaire, if necessary.) 
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11. Does your child have a library card? yes no 



12. About how many times a month does your child go to the library? ( Circle one.) 

0 1 2 3 4 5 more than 5 

13. If your child went to the library, about how many books would you expect her/him to check 
out? 

0 1-3 4-10 more than 10 



14. How much does each of these statements sound like your child? ( Circle a number.) 

a. He/she always reads the same kind of reading material on the same topic (for example, 
just comic books about the same character or just books about horses). 

Not at all Very much 

like my child 1 2 3 4 5 like my child 

b. He/she reads one type of reading material (for example, just books or just newspapers), 
but reads about a variety of topics. 

Not at all Very much 

like my child 1 2 3 4 5 like my child 

c. He/she reads many types of reading materials (for example, books, magazines, 
newspapers, encyclopedias, etc.), but reads mostly on one topic. 

Not at all Very much 

like my child 1 2 3 4 5 like my child 

d. He/she reads many types of reading materials on many different topics. 

Not at all Very much 

like my child 1 2 3 4 5 like my child 
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15. Do you subscribe to any magazines for children? yes no 

If so, how many? ( Circle one.) 

1 2 3 more than 3 



16. Most of the adults that my child knows outside of school ( Check one.) 

hardly ever read for enjoyment. read very often for enjoyment. 

read once in a while for enjoyment. I don’t know. 

read often for enjoyment. 



17. Most of the children that my child plays with outside of school ( Check one.) 

hardly ever read for enjoyment. read very often for enjoyment. 

read once in a while for enjoyment. I don’t know. 

read often for enjoyment. 



*18. During the past school year, how often has your child talked about using the computer for 
reading at school? ( Check one.) 

never occasionally 

often almost everyday 



*19. During the past school year, how has your child’s out-of-school reading changed? 
( Check one.) 

reading much less than before 

reading a little less than before 

reading about the same amount as before 

reading a little more than before 

reading a lot more than before 
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*20. During the past year, how has your child’s interest in reading changed? ( Check one.) 

much less interested in reading 

a little less interested in reading 

about the same interest in reading 

a little more interested in reading 

much more interested in reading 



*21. Comment, if you wish, on your child’s reading, use of the computer, and so forth. We are 
especially interested if you believe you have seen any connection between reading activities 
on the computer at school and reading-related activities outside of school. 



*These items were on the end-of-year survey only. 
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